Sorting with automatic learning

Abstract

The invention relates to a method for sorting articles in a plurality of sorting devices, and a computer readable storage medium storing instructions for implementing the method, in which the method comprises the following steps: sorting (S1) a stream of articles (2) at a first sorting device (1) into at least one sorted portion (3) and optionally at least one unsorted portion (4); capturing (S2) a set of partial images; marking (S3) the partial image with a partial label; at least one classifier configured to correlate the item with the portion is constructed (S4), the at least one classifier being a machine learning model. Further, at the second sorting device (10), capturing (S5) an image of the items in the item stream (12); sorting of the articles in the stream of articles (12) into at least one sorted portion (13) and optionally at least one unsorted portion (14) is controlled (S6) using the images captured at the second sorting device (10) and at least one classifier.

Description

TECHNICAL FIELD

[0001] The present inventive concept relates generally to a method of controlling sorting of material at multiple sorting devices. BACKGROUND

[0002] Existing sorting devices sort material into different parts. The sorting process can involve manual sorting steps, physical sorting steps, and optical sorting steps, and combinations of these steps. For example, in textile sorting, it can be performed to manually sort a stream of garments into a salable part, such as "fashion jeans", etc.

[0003] These processes, especially when they involve more complex selections, are usually customised and optimised for local conditions. That is, setting up a new sorting device properly is costly and time consuming.

[0004] There is therefore a need for a more time- and cost-efficient method of setting up a new sorting device. SUMMARY

[0005] To achieve at least one of the above objects and other objects apparent from the following description, according to the present inventive concept, a method having the features defined in claim 1 is provided. Preferred variants will be apparent from the dependent claims.

[0006] According to a first aspect of the present invention, there is provided a method of controlling sorting of items at multiple sorting devices, the method comprising, at a first sorting device: sorting a stream of items into at least one sorted part and optionally at least one unsorted part; capturing a set of part images of the items in the at least one sorted part; labelling the part images captured at the first sorting device with part labels, wherein each part label indicates a sorted part in the at least one sorted part into which the depicted item has been sorted; constructing at least one classifier configured to relate items to parts, the at least one classifier being a machine learning model, and wherein constructing the at least one classifier comprises: training the machine learning model using the part images captured at the first sorting device and the part labels; at a second sorting device: capturing images of items in the stream of items; using the images captured at the second sorting device and the at least one classifier to control sorting of the items in the stream of items into the at least one sorted part and optionally the at least one unsorted part.

[0007] Thereby, a method of automating and simplifying the sorting process at multiple sorting devices is provided. Since the classifier is constructed by observing the at least one sorted part only, without any information about the internal workings of any actual sorting step, a more cost- and time-efficient technical solution is provided.

[0008] The method can be used for any type of sorting process. That is, the method can be implemented for any type of sorting device and any type of article stream. The article stream can thus comprise any type of article or any combination of articles, such as clothing, minerals, food, empty containers such as metal cans, etc. That is, the articles in the article stream can be made of or comprise plastic, metal, textile material, organic material, mineral material, etc.

[0009] In the context of the present disclosure, the term "sorting device" is thus to be understood as a system performing a sorting process at a particular location. For example, the sorting device can comprise a conveyor belt transporting an article stream on which an operator performing a manual sorting process is located. Alternatively, the sorting device can comprise a conveyor belt transporting an article stream which is monitored by an automatic sorting apparatus. Thus, the first sorting device and the second sorting device are to be understood as being spatially separated, i.e. sorting different article streams. The first sorting device can be arranged at a first sorting plant, while the second sorting device can be arranged at a second sorting plant. Alternatively, the first sorting device and the second sorting device can be arranged at the same sorting plant. Thus, the method can be implemented for sorting devices located at different parts of a plant, at different parts of a city, at different parts of a country, in different countries, and / or on different continents.

[0010] At the first and / or the second sorting device, the respective article stream can be sorted into one or more sorted portions. The respective article stream can for example be sorted into a first and a second sorted portion. Alternatively, the article stream can be sorted into a different number of sorted portions. That is, the article stream at the first sorting device can be sorted into two sorted portions, while the article stream at the second sorting device can be sorted into one sorted portion. That is, even if the classifier is built using two sorted portions, the classifier can be distributed to another sorting device and implemented in that sorting device for sorting the article stream into only one sorted portion.

[0011] In the context of the present disclosure, the term "unsorted portion" is to be understood as a portion whose content is not purposefully selected. That is, the at least one unsorted portion comprises articles which are not sorted into the at least one sorted portion. The article stream can for example be clothing comprising trousers and shirts of different colors, wherein the sorted portion consists of red shirts. Thus, the unsorted portion comprises non-red shirts as well as trousers of different colors. Such unsorted portion can thus constitute articles to be discarded from the sorting process. Additionally or alternatively, the unsorted portion can be recycled into the sorting device or provided to any other sorting device.

[0012] In an example sorting apparatus, the stream of articles is provided in the form of a stream of garments including trousers of different colours, wherein trousers of a first colour are sorted into a first sorted portion and trousers of a second colour are sorted into a second sorted portion. Optionally, any other colour of garment other than the first and second colours will be sorted into an unsorted portion. This unsorted portion can be provided to a further sorting apparatus, for example, for sorting into sorted portions.

[0013] In another example sorting apparatus, the stream of articles is provided in the form of a stream of garments of different types, wherein trousers are sorted into a first sorted portion and shirts are sorted into a second sorted portion. Optionally, any other type of garment other than trousers and shirts can be sorted into an unsorted portion. This unsorted portion can be provided to a further sorting apparatus, for example, for sorting into sorted portions.

[0014] In yet another example sorting apparatus, the stream of articles is provided in the form of objects made of different materials, wherein plastic objects are sorted into a first sorted portion and metal objects are sorted into a second sorted portion. Optionally, objects made of any other material can be sorted into an unsorted portion. This unsorted portion can be provided to a further sorting apparatus, for example, for sorting into sorted portions, or discarded entirely, for example, by dumping or incineration.

[0015] An article being“depicted” in the context of the present disclosure should not be understood to mean that the article must have been imaged by detecting radiation within the visible wavelength band. That is, an article can be“depicted” using radiation of any wavelength, such as infrared, ultraviolet, X-rays, etc.

[0016] Capturing a set of partial images of the articles in the at least one sorted portion can be performed at any point in time after the articles have been separated from the stream of articles. For example, if the sorting apparatus comprises a conveyor belt for providing the stream of articles to a physical sorting location, wherein each sorted portion is transported on a separate conveyor belt after the physical sorting step, the set of partial images can be captured at any moment in time when the articles of the sorted portion are located on the separate conveyor belt. As another example, the set of partial images can be captured when the articles in the sorted portion have been separated from the stream of articles but are in a stationary state.

[0017] It will be appreciated that a set of partial images can comprise any non-zero number of images. For example, a set of partial images can comprise one or more images, such as 10 images, or 100 images, or 1000 images, or 10000 images.

[0018] More than one partial image can be taken of any one item. Thus, an item can be present in more than one image. Furthermore, more than one item can be present in any one image. It is not required for the purposes of the inventive concept that any one object can be tracked. Rather, the purpose of the set of partial images is to train a classifier, which is a machine learning model.

[0019] The images can be taken by any type of camera arrangement known in the art.

[0020] The machine learning model can be any type of machine learning model. For example, the machine learning model can be a neural network, such as a convolutional neural network or a fully connected neural network. The machine learning model can be a neural network with any number of input nodes, any number of hidden layers and corresponding nodes in these layers, and any number of output nodes. Furthermore, any node of the neural network can have any type of activation function, such as a rectified linear unit (ReLU) activation function, or a logistic function.

[0021] As is well known in the art, an image label is to be understood as information attached to or associated with an image and relates to any feature depicted by the image, or any context in which the image was taken, etc., which can be important for training the machine learning model. Here, the label indicates the sorted portion in which the depicted item has been sorted to. That is, the label can for example be a keyword, such as “green”, or “pants”, or “aluminum”. An image can have more than one label attached to it or associated with it. The label can be a multi-class label, i.e. a label containing more than one element. That is, the label can be a vector containing multiple elements. For example, the label can be a multi-class label containing the elements “pants”, “shorts”, and “green”.

[0022] Labeling the partial images can further comprise, for each partial image, identifying at least one depicted item, wherein the partial label further indicates each identified item.

[0023] For example, identifying the depicted item can comprise determining a class or type of the item. For example, identifying the depicted item can comprise determining that the item is of the type “pants”, or “shirt”, or “red pants”, or “blue shirt”, or “can”, or “aluminum can”, etc. Furthermore, it is to be understood that a partial image can depict more than one item. Thus, labeling the partial images can comprise identifying more than one item for any one image.

[0024] Identifying the depicted item can be performed in a variety of ways. For example, identifying the depicted item can be performed by foreground-background segmentation. Alternatively, identifying the depicted item can be performed by object instance segmentation.

[0025] If any items in the item flow are stacked, object instance segmentation is particularly useful because it can be used to distinguish individual items within a stack.

[0026] Object instance segmentation can be performed by a machine learning model. For example, this machine learning model can be part of a classifier, or the same machine learning model as the classifier. Therefore, object instance segmentation can be performed by a neural network, such as a convolutional neural network or a fully connected neural network. After an image has been processed by a machine learning model using object instance segmentation, that image can be fed as training material to the classifier.

[0027] Images in this set of images may undergo further image processing steps. For example, images in this set of images may undergo image enhancement, cropping, etc. Furthermore, it should be understood that captured portions of images may be discarded from the set, for example, if no objects can be identified, or if the image quality is too low.

[0028] The method may further include: extracting a set of partial object images from the set of partial images, each partial object image depicting a recognized object, such that the number of partial object images in the set of partial object images is the same as the number of objects recognized in the set of partial images; and wherein labeling may be performed on the set of partial object images, and wherein partial object images and partial labels captured at the first sorting device may be used to train a machine learning model.

[0029] Therefore, if an image depicts more than one item, a set of item images can be extracted from the image, resulting in a cleaner training dataset. Furthermore, the partial item images can have a smaller data size than the original partial images. This provides a more easily processed set of images.

[0030] It should be noted that any one of the item images in this set can be discarded.

[0031] The method may further include: at a first sorting device; capturing a set of stream images of items in the stream before sorting the stream of items into at least one sorted portion and optionally at least one unsorted portion; and labeling the stream images with stream tags, wherein the labeling may include: identifying at least one depicted item for each stream image, wherein the stream tag may also indicate each identified item.

[0032] It should be understood that a stream of images can include any non-zero number of images. For example, a stream of images can include one or more images, such as 10 images, or 100 images, or 1000, or 10000 images.

[0033] This set of streaming images can be processed as described above regarding the partial images in this set. Specifically, the images in this set of streaming images can undergo further image processing steps. For example, the images in this set of streaming images can undergo image enhancement, cropping, etc. Furthermore, it should be understood that captured streaming images can be discarded from this set, for example, if no objects can be identified, or if the image quality is too low.

[0034] Because streaming images can depict first and second items that are substantially different from each other in classification, such as "blue jeans" and "green shirt," or "aluminum can" and "plastic bottle," they can be provided with more than one label, or multiple category labels.

[0035] The labeled stream images may further include: extracting a set of stream item images from the set of stream images, each stream item image depicting an identified item, such that the number of stream item images in the set of stream item images may be the same as the number of items identified in the set of stream images; and wherein labeling may be performed on the set of stream item images.

[0036] This set of streaming images or this set of streaming item images can be used in a variety of ways.

[0037] The method may further include: comparing the set of streaming images with the set of partial images, or the set of streaming item images with the set of partial item images, and removing images of items depicted in any of the partial images or the set of streaming item images from the set of streaming images or the set of streaming item images, thereby generating a modified set of streaming images or a modified set of streaming item images; labeling each streaming image in the modified set of streaming images or each streaming item image in the modified set of streaming item images with a label indicating that the depicted item has been sorted into at least one unsorted portion; and wherein constructing at least one classifier may further include: training a machine learning model using the streaming images in the modified set of streaming images or the streaming item images in the modified set of streaming item images captured at the first sorting device and each associated label.

[0038] Therefore, the machine learning model is trained on a larger dataset and is trained not only to distinguish items to be sorted into the already sorted portion, but also to distinguish items not sorted into such a sorted portion. This results in a more accurate classifier.

[0039] Marking parts of an image can be done in several ways.

[0040] Preferably, the use of partial labels to mark a portion of the image captured at the first sorting device is performed by the processing unit, and wherein the processing unit is further configured to provide the partial image and partial labels captured at the first sorting device to a machine learning model.

[0041] It should be noted that the above can be applied to the labeling of any other group of images, such as the group of streaming images or modified streaming images. That is, labeling streaming images or modified streaming images can be performed by the processing unit, preferably by the same processing unit that labels the portion of the images.

[0042] The processing unit may be a central processing unit. Alternatively, the processing unit may include multiple components arranged separately and communicatively connected to each other. The processing unit may also be configured to perform the following functions: collect and compile the set of partial images and / or streaming images, compare the set of streaming images with the set of partial images, and remove images depicting objects described in any of the partial images from the set of streaming images, thereby producing a modified set of streaming images. Furthermore, the processing unit may be configured to provide a machine learning model with the streaming images from the modified set of streaming images captured at the first sorting device and each associated label. Additionally, the processing unit may be configured to perform the function of rejecting anomalous images (i.e., images that deviate from the rest of the set of images by more than a predetermined permissible level). The rejection of anomalous images may optionally be supplemented by manual operator intervention.

[0043] The classifier can be distributed to the second sorting device in several ways. For example, the classifier can be stored on a non-transitory storage medium at the first sorting device, which is then transported to the second sorting device.

[0044] Preferably, the method further includes distributing at least one classifier from the first sorting device to the second sorting device via a network infrastructure such as a cloud-based server.

[0045] Therefore, the distribution of at least one classifier can be automated.

[0046] At the second sorting device, the classifier can be implemented, for example, on a controller unit. Therefore, this controller unit can be arranged to communicate with the first sorting device via the same network infrastructure, for example, by communicating with the same cloud-based server.

[0047] The method may further include: updating at least one classifier distributed to a second sorting device, wherein updating at least one classifier distributed to the second sorting device includes: at the first sorting device: repeating the following steps: sorting the item stream into at least one sorted portion and optionally at least one unsorted portion; capturing a set of partial images of items in at least one sorted portion; labeling the partial images captured at the first sorting device with partial labels; and constructing at least one classifier, including: training a machine learning model using the partial images captured at the first sorting device and the partial labels to provide at least one updated classifier; distributing the at least one updated classifier to the second sorting device; and activating the at least one updated classifier at the second sorting device.

[0048] Therefore, the classifier is first updated at the first sorting unit by further learning through a machine learning model, and then the updated classifier is distributed to other sorting units. Since the classifier only needs to be updated centrally, i.e., at the first sorting unit, this provides a more efficient method.

[0049] Activating at least one updated classifier at the second sorting device can, for example, mean deactivating at least one earlier version of the classifier. That is, the updated classifier replaces the earlier classifier only at the functional level, i.e., as the classifier controlling the sorting process. Thus, the earlier classifier can still be retained in the system and can be selectively activated when needed. Therefore, a more flexible technical solution is provided.

[0050] Updating at least one classifier distributed to the second sorting device may further include: sending a verification request from the second sorting device to the first sorting device, wherein if the verification request fails, the following steps are performed: at the first sorting device: repeating the following steps: sorting the item stream into at least one sorted portion and optionally at least one unsorted portion; capturing a set of partial images of items in at least one sorted portion; labeling the partial images captured at the first sorting device with partial labels; and constructing at least one classifier, including: training a machine learning model using the partial images captured at the first sorting device and the partial labels to provide at least one updated classifier; distributing the at least one updated classifier to the second sorting device; and activating the at least one updated classifier at the second sorting device.

[0051] The verification request may include a request to compare the machine model version at the first sorting unit with the machine model version at the second sorting unit. Therefore, if it is determined that the versions are not the same, the verification request can be made to fail.

[0052] Alternatively, the verification request can be sent from the first sorting device to any other sorting device. That is, an update from the first sorting device can be pushed to another sorting device.

[0053] Training the machine learning model may also include: embedding data of partial images and associated partial labels captured at the first sorting device, and optionally data of stream images and associated stream labels captured at the first sorting device, as data points into a spatial graph, such as a two-dimensional or three-dimensional graph; and dividing the spatial graph into cells, each cell having a data point density and a last entry date indicating the time when the latest data point was added to the cell.

[0054] The data point density of a cell can also be referred to as embedding density. Embedded data of a set of images can be useful for a variety of reasons. For example, repeating the steps of building at least one classifier may also include: for each cell, checking the data point density and / or the last entry date, and if the cell's data density is equal to or higher than a predetermined threshold, and / or if the cell's last entry date is equal to or earlier than a predetermined threshold, modifying the set of data points contained within the cell, such as removing each data point from that set of data points.

[0055] In other words, if a cell has too high a data point density, or if the last entry date is too old, the set of data points contained within the cell can be modified accordingly. For example, if the last entry date is too old, meaning too much time has passed since the data points were added to the cell, all data points in the cell can be removed. This can be beneficial, for example, if the data points in the cell represent items that no longer need to be sorted.

[0056] Alternatively or additionally, repeating the step of constructing at least one classifier may also include: when adding a new data point to the spatial graph, checking the data point density and / or the last entry date of the cell to which the data point will be added, and: if the data density of the cell is equal to or higher than a predetermined threshold, discarding the new data point; and / or if the last entry date of the cell is equal to or earlier than the predetermined threshold, adding the new data point and removing at least one adjacent data point in the cell.

[0057] Therefore, when the training dataset is expanded, redundant or outdated data points can be discarded. Thus, at least one classifier can be updated to control the sorting of new items, or to control the sorting of items in a new way.

[0058] The predetermined threshold for the last entry date can be, for example, one week, one month, several months, one year, or several years.

[0059] The predetermined threshold for data density can be, for example, a corresponding number assigned to a cell, such as 10, 20, or 100, indicating that the cell can contain only 10, 20, or 100 data points. All cells can have the same threshold. A cell can have a different threshold than other cells.

[0060] Alternatively, the predetermined threshold for a unit can be set according to a specified ratio, which is calculated by dividing the total number of data points in all units by the number of units. Therefore, the predetermined threshold for a unit could be 10 data points per unit, 20 data points per unit, or 100 data points per unit. Thus, the exact value can depend on the total number of data points in the dataset when the predetermined threshold is set. Furthermore, the predetermined threshold can change over time. The predetermined threshold can be refreshed, for example, on each update of at least one classifier, such that each unit is provided with an updated ratio.

[0061] Alternatively, the predetermined threshold can be set based on the probability distribution of data points on the spatial graph. That is, a cell that frequently adds data points or already includes a large number of data points can have a higher predetermined threshold than a cell that adds data points less frequently or includes fewer data points. Thus, when a cell assigned a low probability density value is provided with new data points, each other cell, or at least each other cell with a higher probability density value, can increase its predetermined threshold.

[0062] Alternatively or concurrently, the dataset can be cleaned before adding any additional data points, particularly when updating at least one classifier. For example, the dataset can be cleaned according to a specific ratio for each unit. For instance, 5%, 10%, 20%, or 30% of all data points in each unit can be removed before any additional data points can be provided to any unit.

[0063] Any number of classifiers can be constructed. For example, the method according to any of the preceding claims, wherein constructing at least one classifier comprises: constructing a corresponding classifier for each of the at least one sorted portion. Alternatively, only one classifier may be constructed, wherein the classifier at the second sorting device is configured to sort the item flow into one or more sorted portions. That is, one classifier is configured to control the sorting of the item flow into multiple sorted portions. Alternatively, more than one classifier may be constructed. For example, one classifier may be constructed for each sorted portion, wherein at the second sorting device, the sorting of the item flow into multiple sorted portions is controlled by multiple classifiers. As an example, two classifiers may be constructed at the first sorting device, such that at the second sorting device, the two classifiers control the sorting of the item flow into the corresponding sorted portions.

[0064] Any number of classifiers can be distributed to other sorting devices. That is, the number of classifiers built at the first sorting device does not have to be equal to the number of classifiers distributed to a specific sorting device. For example, two classifiers can be built at the first sorting device, and only one classifier can be distributed to the second sorting device.

[0065] Furthermore, the classifier can be distributed to more than one additional sorting device. Additionally, the number of classifiers distributed to the second sorting device can differ from the number distributed to the third sorting device.

[0066] That is, the method may further include: at the third sorting device: capturing images of items in the item stream; using the images captured at the third sorting device and a classifier to control the sorting of items in the item stream into at least one sorted portion and optionally at least one unsorted portion.

[0067] At any additional sorting device (i.e., a second or third sorting device), the sorting process can be configured to take into account the classifier to be distributed to that sorting device. In other words, the additional sorting device can be built from scratch based on the classifier.

[0068] According to a second aspect of the present invention, a computer-readable storage medium is provided that stores instructions for implementing the method according to the first aspect of the present invention.

[0069] Any benefits or technical effects discussed in connection with the first aspect of the present invention can be applied to the second aspect of the present invention, and therefore will not be mentioned again to avoid unnecessary repetition. Attached Figure Description

[0070] The concept of the invention will be described in more detail below with reference to the accompanying drawings, in which: Figure 1 This is a schematic diagram of a system for implementing a method according to a first aspect of the concept of the present invention, wherein a first sorting device and a second sorting device are shown, the first sorting device being communicatively connected to a processing unit, and the second sorting device including a controller being communicatively connected to the processing unit of the first sorting device; Figure 2 This is a flowchart illustrating a method according to a first aspect of the concept of the present invention; Figures 3A-3B It is a schematic diagram of the spatial embedding of data; and Figure 4 This is a perspective view of an exemplary second sorting device for sorting textiles. Detailed Implementation

[0071] The inventive concept will be described more fully below with reference to the accompanying drawings, in which preferred variations of the inventive concept are illustrated. However, the inventive concept can be varied in many different ways and should not be construed as limited to the variations set forth herein; rather, these variations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concept to those skilled in the art. In the drawings, the same reference numerals denote the same elements.

[0072] exist Figure 1 The diagram schematically illustrates the sorting process at the first sorting device 1. Here, the input stream of items 2 is provided to the first sorting device 1.

[0073] The sorting device 1 can perform manual sorting, automatic sorting, or any combination thereof. For example, the sorting device 1 may include a conveyor belt of the incoming goods flow 2 located at which an operator performing the manual sorting process is situated. Alternatively, the sorting device 1 may include a conveyor belt of the incoming goods flow 2 monitored by automatic sorting equipment. Exemplary sorting devices in... Figure 4 Shown and associated Figure 4 discuss.

[0074] Item flow 2 can include any type of item or any combination of items, such as clothing, minerals, food, empty containers (such as metal cans), etc. That is, items in item flow 2 can be made of or include these materials, such as plastics, metals, textiles, organic materials, minerals, etc.

[0075] At the first sorting device 1, the item flow 2 is sorted into at least one sorted section 3. Here, the item flow 2 is sorted into the first and second sorted sections 3. However, the item flow 2 can be sorted into any number of sorted sections, i.e., into sorted sections 1 to N, where N can be any number. Figure 1 As shown, the item flow 2 is also sorted into an optional unsorted portion 4. The unsorted portion 4 may consist of items to be discarded from the sorting process. Alternatively or alternatively, the unsorted portion 4 may be recycled back to the sorting device 1 or provided to any other sorting device.

[0076] In the exemplary sorting device 1, the item flow 2 is provided in the form of a garment flow including trousers of different colors, wherein trousers of a first color are sorted into a first sorted section 3, and trousers of a second color are sorted into a second sorted section 3. Optionally, garments of any other color besides the first and second colors will be sorted into an unsorted section 4.

[0077] In another exemplary sorting device 1, the item flow 2 is provided in the form of different types of garment flow, wherein trousers are sorted into a first sorted section 3, and shirts are sorted into a second sorted section 3. Optionally, any other type of garment besides trousers and shirts can be sorted into an unsorted section 4.

[0078] In another exemplary sorting device 1, the article flow 2 is provided in the form of objects made of different materials, wherein plastic objects are sorted into a first sorted section 3, and metal objects are sorted into a second sorted section 3. Optionally, objects made of any other material may be sorted into an unsorted section 4.

[0079] like Figure 1 As further shown, camera device 5 is observing two sorted sections 3. Camera device 5 can include any camera known in the prior art. Camera device 5 is configured to capture a set of partial images of items in the first and second sorted sections 3. The capture of partial images can be performed at any point in time after the items have been separated from the input item flow 2. For example, if the sorting device 1 includes a conveyor belt for providing the item flow 2 to the physical sorting location, wherein each sorted section is transported on a separate conveyor belt after the physical sorting step, a set of partial images can be captured at any moment when the items in the sorted section are on that separate conveyor belt. As another example, a set of partial images can be captured when the items in the sorted section 3 have been separated from the item flow but are stationary.

[0080] It should be understood that this set of partial images can include any non-zero number of images. For example, a set of partial images can include one or more images, such as 10 images, or 100 images, or 1000, or 10000 images. More than one partial image can be taken of any one object. Therefore, an object can exist in more than one image. Furthermore, more than one object can exist in any single image.

[0081] The method may further include extracting a set of partial object images from the set of partial images. Each partial object image may depict a recognized object, such that the number of partial object images in the set of partial object images can be the same as the number of objects recognized in the set of partial images. For example, three object images can be extracted from images depicting three objects. Labeling can be performed on the set of partial object images, and a machine learning model can be trained using the partial object images and partial labels captured at the first sorting device. Thus, if an image depicts more than one object, a set of object images can be extracted from the image, resulting in a cleaner training dataset. Furthermore, the partial object images can have a smaller data size than the original partial images. This provides a set of images that is easier to process.

[0082] The set of partial images or the set of partial object images are provided to the processing unit 6. The processing unit 6 may be a central processing unit. Alternatively, the processing unit 6 may include multiple components arranged separately and communicatively connected to each other. The processing unit 6 is arranged to communicate with the camera device 5 observing the two sorted portions 3. Therefore, the processing unit 6 may also be configured to control the camera device 5 and / or retrieve data such as the set of partial images from the camera device 5.

[0083] Furthermore, here, the processing unit 6 is configured to use partial labels to mark partial images captured at the first sorting device 1. The partial labels indicate the sorted portion 3 to which the depicted item has been sorted. That is, the labels can be, for example, keywords such as "green," "trousers," or "aluminum."

[0084] Processing unit 6 implements a machine learning model. The machine learning model can be of any type. For example, it can be a neural network, such as a convolutional neural network or a fully connected neural network. Processing unit 6 is also configured to provide the machine learning model with partial images and partial labels captured at the first sorting device 1. Thus, processing unit 6 is configured to construct at least one classifier, which is configured to associate items with parts. Constructing at least one classifier includes training the machine learning model using the partial images and partial labels captured at the first sorting device 1.

[0085] An additional camera device 7 is observing the input item flow 2. The additional camera device 7 is configured to capture a set of stream images of the items in the item flow 2 before sorting the item flow 2 into two sorted portions 3 and an optional unsorted portion 4. It should be understood that this set of stream images can include any non-zero number of images. For example, the set of stream images can include one or more images, such as 10 images, 100 images, or 1000 images.

[0086] Here, processing unit 6 is further configured to compare the set of stream images with the set of partial images, and remove images depicting objects in any of the partial images from the set of stream images, thereby generating a modified set of stream images. Furthermore, processing unit 6 is configured to label each stream image in the modified set of stream images with a label indicating that the depicted object has been sorted into the unsorted portion 4. Additionally, constructing at least one classifier includes training a machine learning model using the stream images in the modified set of stream images captured at the first sorting device and each associated label. That is, processing unit 6 is also configured to feed the modified set of stream images to the machine learning model to construct a classifier.

[0087] It should be noted that the steps described above for comparing and removing to produce a modified set of images, as well as the subsequent steps for marking the images in the modified set of images, can also be applied to groups of item images, i.e. groups of partial item images and groups of stream item images.

[0088] Processing unit 6 can also be configured to identify at least one depicted item for each image, wherein a label associated with the image further indicates the identified item. For example, identifying a depicted item may include determining the category or type of the item. For example, identifying a depicted item may include determining that the item belongs to a type such as “trousers,” “shirt,” “red trousers,” “blue shirt,” “can,” or “aluminum can.” Furthermore, it should be understood that a portion of an image may depict more than one item. Therefore, labeling a portion of an image may include identifying more than one item for any given image. Identifying depicted items can be performed in various ways. For example, identifying depicted items can be performed by foreground-background segmentation. Alternatively, identifying depicted items can be performed by object instance segmentation. Object instance segmentation can be performed by a machine learning model. For example, the machine learning model can be part of a classifier, or the same machine learning model as the classifier. Therefore, object instance segmentation can be performed by a neural network, such as a convolutional neural network or a fully connected neural network. After an image has been processed by a machine learning model using object instance segmentation, the image can be provided as training material to the classifier. Images within any set of images may undergo further image processing steps. For example, images in a subset of the set may undergo image enhancement, cropping, etc. Furthermore, it should be understood that captured images may be discarded from the corresponding set, for example, if no objects can be identified, or if the image quality is too low.

[0089] Furthermore, processing unit 6 can be configured to perform the function of rejecting abnormal images (i.e., images that deviate from the rest of the group by more than a predetermined tolerance level). The rejection of abnormal images can optionally be supplemented by manual operator intervention.

[0090] exist Figure 1 The lower half of the diagram shows a second sorting device 10, which sorts the input stream of goods 12. The second sorting device 10 is spatially separated from the first sorting device 1. This should be understood as the first sorting device 1 and the second sorting device 10 sorting different streams of goods 2, 12. The first sorting device 1 can be located at a first sorting plant, while the second sorting device 10 can be located at a second sorting plant. Alternatively, the first sorting device 1 and the second sorting device 10 can be located at the same sorting plant. Therefore, this method can be implemented for sorting devices located in different parts of a factory, different parts of a city, different parts of a country, in different countries, and / or on different continents.

[0091] The second sorting device 10 is communicatively connected to the controller 16. The controller is communicatively connected to the processing unit 6 of the first sorting device 1. The controller 16 can be connected to the processing unit 6 in various ways, such as via network infrastructure, for example, via a communication connection through a cloud-based server. The classifier can therefore be distributed from the processing unit 6 to the controller 16.

[0092] The controller 16 can thus implement a machine learning model of the classifier to control the sorting process of the second sorting device 10. Here, the input item flow 12 is sorted into the first and second sorted portions 13 and optionally the unsorted portion 14. Therefore, the item flow 12 input to the second sorting device 10 is sorted into the same number of sorted portions as the item flow 2 input to the first sorting device 1. Alternatively, item flows 2 and 12 can be sorted into different numbers of sorted portions 3 and 13. That is, the item flow 2 at the first sorting device 1 can be sorted into two sorted portions 3, while the item flow 12 at the second sorting device 10 can be sorted into one sorted portion 13.

[0093] exist Figure 2 The diagram illustrates a flowchart of an exemplary method according to a first aspect of the present invention. At a first sorting device 1, an action is taken to sort a stream of items 2, S1, into at least one sorted portion 3 and optionally at least one unsorted portion 4. Subsequently, an action is taken to capture, S2, a set of partial images of the items in at least one sorted portion 3, followed by an action is taken to label, S3, the partial images captured at the first sorting device 1 using partial tags. Each partial tag indicates the sorted portion 3 to which the depicted item has been sorted. Labeling the partial images, S3, may further include: for each partial image, identifying at least one depicted item, wherein the partial tag further indicates the identified item. Identifying at least one depicted item may be performed, for example, by object instance segmentation.

[0094] Subsequently, the execution of construction S4 is configured to associate items with at least one classifier. The at least one classifier is a machine learning model, and constructing at least one classifier in S4 includes training the machine learning model using partial images and partial labels captured at the first sorting device 1. Constructing at least one classifier in S4 may include constructing a corresponding classifier for each of the at least one sorted portion 3.

[0095] like Figure 2 As shown, the method may optionally include: at the first sorting device 1, before sorting the item flow 2 into at least one sorted portion 3 and optionally at least one unsorted portion 4, capturing S31 a set of flow images of items in the item flow 2. Furthermore, after capturing S31 a set of flow images, comparing the set of flow images with the set of partial images can be performed S32. Thereafter, removing S33 images depicting items in any of the partial images from the set of flow images can be performed, thereby generating a modified set of flow images. Subsequently, labeling each flow image in the modified set of flow images using labels indicating that the depicted items have been sorted into at least one unsorted portion 4 is performed S34. Thus, constructing S5 at least one classifier may further include: training a machine learning model using the flow images in the modified set of flow images captured at the first sorting device 1 and each associated label.

[0096] The method further includes, at the second sorting device 10: capturing images of items in the item stream 12 in S5; and using the images captured at the second sorting device 10 and at least one classifier to control S6 to sort the items in the item stream 12 into at least one sorted portion 13 and optionally at least one unsorted portion 14.

[0097] The method may also include distributing at least one classifier from the first sorting device 1 to the second sorting device 10 via a network infrastructure such as a cloud-based server.

[0098] The method may further include updating at least one classifier distributed to the second sorting device 10 in step S8. Updating at least one classifier distributed to the second sorting device in step S8 includes: at the first sorting device 1, repeating the steps of sorting S1, capturing S2, marking S3, and constructing S4 to provide at least one updated classifier. Furthermore, updating S8 includes: distributing at least one updated classifier to the second sorting device 10, and activating at least one updated classifier at the second sorting device 10.

[0099] Updating S8 to at least one classifier distributed to the second sorting device 10 may further include: sending a verification request from the second sorting device 10 to the first sorting device 1, wherein if the verification request fails, the steps of sorting S1, capturing S2, marking S3 and constructing S4 are performed.

[0100] Updating at least one classifier in S8 may further include training a machine learning model by selectively forgetting data. This can be achieved in various ways. For example, training the machine learning model may also include embedding data of partial images and associated partial labels captured at the first sorting device, and optionally data of stream images and associated stream labels captured at the first sorting device, as data points into a spatial graph, such as a two-dimensional or three-dimensional graph; and dividing the spatial graph into cells, each cell having a data point density and a last entry date indicating the time when the latest data point was added to the cell. The data point density of a cell may also be referred to as the embedding density. An exemplary two-dimensional spatial embedding graph 20 is shown in Figure 20. Figures 3A-3B As shown in the image.

[0101] Embedded data from a set of images can be useful for a variety of reasons. For example, repeating the step of building at least one classifier may also include: for each cell, checking the data point density and / or the last entry date, and if the cell's data density is equal to or higher than a predetermined threshold, and / or if the cell's last entry date is equal to or earlier than a predetermined threshold, modifying the set of data points contained within the cell, such as removing each data point from that set of data points. Alternatively or additionally, repeating the step of building at least one classifier may also include: when adding new data points to the spatial map, checking the data point density and / or the last entry date of the cell to which the data point is to be added, and: if the cell's data density is equal to or higher than a predetermined threshold, discarding the new data point; and / or if the cell's last entry date is equal to or earlier than a predetermined threshold, adding the new data point and removing at least one neighboring data point in the cell.

[0102] Therefore, when the training dataset is expanded, redundant or outdated data points can be discarded. Thus, at least one classifier can be updated to control the sorting of new items, or to control the sorting of items in a new way.

[0103] It should be understood that the above method can be implemented for more than one additional sorting device. For example, the following steps can be performed: at the third sorting device: capturing images of items in the item flow; using the images captured at the third sorting device and a classifier to control the sorting of items in the item flow into at least one sorted portion and optionally at least one unsorted portion.

[0104] Instructions for implementing the exemplary methods described above, and any variations thereof, may be stored on a computer-readable storage medium. Processing unit 6 may thereby utilize these instructions to execute the method.

[0105] exist Figure 3A The figure illustrates the spatial embedding of data as data point 21 in a two-dimensional spatial diagram 20. The data can be derived, for example, from a set of partial images or a set of flow images, or a set of partial object images or a set of flow object images. That is, data point 21 can represent an object depicted in a partial image and / or a flow image. In other words, data point 21 can additionally or alternatively represent an object depicted in an object image (such as a partial object image or a flow object image). Therefore, the placement of data point 21 in Figure 20 can be determined according to the category to which the object belongs. For this purpose, data points can be arranged into clusters 22, where each cluster 22 thus represents a class of objects. Figure 3A In the data, data points 21 belonging to the same cluster 22 are indicated by the same grayscale shading.

[0106] exist Figure 3B In Figure 20, a grid has been applied. The grid divides Figure 20 into cells 23. Each cell 23 has a data point density. Furthermore, each cell 23 may have an associated last entry date, indicating when the latest data point was added to cell 23. Repeating the steps of constructing at least one classifier may also include: for each cell 23, checking the data point density and / or the last entry date, and if the data density of cell 23 is equal to or higher than a predetermined threshold, and / or if the last entry date of cell 23 is equal to or earlier than a predetermined threshold, modifying a set of data points 21 contained within cell 23, such as removing each data point 21 from that set of data points 21. That is, if cell 23 has too high a data point density, or if the last entry date is too old, the set of data points 21 contained within the cell can be modified accordingly. For example, if the last entry date is too old, i.e., too much time has passed since data point 21 was added to cell 23, all data points 21 in cell 23 can be removed. For example, it could be beneficial if the data points in a cell represent items that no longer need to be sorted.

[0107] Alternatively or additionally, repeating the step of constructing at least one classifier may also include: when adding a new data point 21 to the spatial graph 20, checking the data point density and / or the last entry date of the cell 23 to which the data point 21 is to be added, and: if the data density of the cell 23 is equal to or higher than a predetermined threshold, discarding the new data point 21; and / or if the last entry date of the cell 23 is equal to or earlier than the predetermined threshold, adding the new data point 21 and removing at least one adjacent data point 21 in the cell 23.

[0108] The predetermined threshold for the last entry date can be, for example, one week, one month, several months, one year, or several years.

[0109] The predetermined threshold for data density can be, for example, a corresponding number assigned to cell 23, such as 10, 20, or 100, indicating that cell 23 can contain only 10, 20, or 100 data points. All cells 23 can have the same threshold. Cell 23 can have a different threshold than other cells 23.

[0110] Alternatively, the predetermined threshold for unit 23 can be set according to a specified ratio, which is calculated by dividing the total number of data points 21 in all units 23 by the number of units 23. Therefore, the predetermined threshold for a unit can be 10 data points 21 per unit, 20 data points 21 per unit 23, or 100 data points 21 per unit 23. Thus, the exact value can depend on the total number of data points 21 included in the dataset when the predetermined threshold is set. Furthermore, the predetermined threshold can change over time. The predetermined threshold can be refreshed, for example, on each update of at least one classifier, such that each unit 23 is provided with an updated ratio.

[0111] Alternatively, the predetermined threshold can be set based on the probability distribution of data points 21 on the spatial graph 20. That is, a unit 23 that frequently adds data points 21 or already includes a large number of data points 21 can have a higher predetermined threshold than a unit 23 that adds data points 21 less frequently or includes fewer data points. Thus, when a unit 23 assigned a low probability density value is provided with a new data point 21, each other unit 23, or at least each other unit 23 with a higher probability density value, can increase its predetermined threshold.

[0112] Alternatively or concurrently, the dataset can be cleaned before adding any additional data points 21, particularly when updating at least one classifier. For example, the dataset can be cleaned at a specific ratio for each unit 23. For instance, 5%, 10%, 20%, or 30% of all data points 21 in each unit 23 can be removed before any additional data points 21 can be provided to any unit.

[0113] Figure 4 A perspective view of an exemplary second sorting device 700 implementing at least one classifier is shown, wherein at least one classifier has been distributed to the second sorting device 700.

[0114] The sorting device 700 is supplied with a flow of articles 710, such as textile materials, like clothing. The materials 710 are conveyed through the inspection zone 720 via a transporter (such as a conveyor belt or chute). However, materials can be provided for passage through the inspection zone 720 by any suitable means or manually without any technical means. Inside the housing 222 are a light source device 730 and an inspection system 740, which includes a camera system 750 and optionally a spectrometer system 760. The camera system 750 and the optional spectrometer system 760 are adapted to receive and analyze light emitted by the light source device 730 and subsequently reflected and / or scattered by the materials in the inspection zone 720. The light source device 730 typically emits spectra in the UV, VIS, and / or NIR wavelength range. The spectrometer system 760 typically acquires spectra in the UV, VIS, and / or NIR wavelength range; and the camera system 750 typically acquires images in the UV, VIS, and / or NIR wavelength range. The inspection system 740 of the sorting device 700 is configured to distinguish a group of textile materials from other materials based on the acquired spectrum and / or the acquired images. In other words, the system 740 can be set up so that a particular type of material or a group of materials can be distinguished from other types of materials based on its spectrum (including its color) and / or based on its shape, size, or any other detectable appearance.

[0115] The sorting device 700 may optionally include a laser triangulation system 746 configured to determine height information related to the material being transported through the detection zone 720.

[0116] Furthermore, the sorting device 700 preferably includes an exhaust system 224 (such as a robotic arm or a nozzle for exhausting pressurized air) for sorting materials into different portions based on images captured by the camera system and at least one classifier. That is, at least one classifier can be implemented on a controller 16 arranged in the sorting device 700, which is also configured to perform functions such as physically sorting items, such as actuating the robotic arm or nozzle. Thus, such a controller 16 can be communicatively connected to the light source device 730 and an inspection system including a camera system 750 and optionally a spectrometer system 760, and can also be configured to actuate these systems. Additionally or alternatively, information received from the inspection system and optionally a triangulation system can also be used to verify sorting decisions made based on the classifier, or to enable sorting in more refined portions. Therefore, images captured at the second sorting device 700 and at least one classifier are used when sorting the stream of items into at least one sorted portion. A more detailed description of the sorting device 700 (including the light source device, inspection system, laser triangulation system and discharge system 224) can be found in WO 2015 / 063300, which is incorporated herein by reference.

[0117] Preferred variations and examples of the inventive concept have been disclosed in the accompanying drawings and description, and although specific terminology has been used, it is used only in a general and descriptive sense and not for the purpose of limiting the scope of the inventive concept set forth in the following claims.

Claims

1. A method for controlling the sorting of articles in multiple sorting devices, the method comprising: At the first sorting device (1): The flow of items (2) is sorted (S1) into at least one sorted portion (3) and optionally at least one unsorted portion (4); Capture (S2) a set of partial images of the items in at least one sorted section (3); The partial image captured at the first sorting device (1) is marked (S3) using partial labels, wherein each partial label indicates the sorted portion (3) to which the depicted item has been sorted. The construction (S4) is configured to associate an item with a part, the at least one classifier being a machine learning model, and wherein constructing the at least one classifier includes training the machine learning model using the part image captured at the first sorting device and the part label; At the second sorting device (10): Capture (S5) images of items in the item stream (12); Using the image captured at the second sorting device (10) and the at least one classifier, control (S6) sort the items in the item stream (12) into at least one sorted portion (13) and optionally at least one unsorted portion (14).

2. The method according to claim 1, wherein, The marker (S3) also includes, for each part of the image: Such as identifying at least one depicted item by object instance segmentation, wherein a corresponding partial label further indicates each identified item.

3. The method according to claim 2, further comprising: From the set of partial images, a set of partial object images is extracted, each partial object image depicting a recognized object, such that the number of partial object images in the set of partial object images is the same as the number of recognized objects in the set of partial images; and In this process, labeling (S3) is performed on the set of partial item images, and the machine learning model is trained using the partial item images captured at the first sorting device and the partial labels.

4. The method according to claim 2 or 3, further comprising: At the first sorting device; Before sorting the item flow (2) into at least one sorted portion (3) and optionally at least one unsorted portion (4), a set of flow images of the items in the item flow (2) is captured (S31); as well as The streaming images are labeled using streaming tags (S32), wherein labeling (S32) includes: for each streaming image, identifying at least one depicted item, wherein the streaming tag further indicates each identified item.

5. The method according to claim 4, wherein, The marker (S32) also includes: From the set of streaming images, extract a set of streaming item images, each streaming item image depicting a recognized item, such that the number of streaming item images in the set of streaming item images is the same as the number of items recognized in the set of streaming images; and In this process, marking is performed on the set of streaming item images (S32).

6. The method according to claim 4 or 5, further comprising: The set of streaming images is compared with the set of partial images, or the set of streaming item images is compared with the set of partial item images (S33), and images of items depicted in any of the partial images or the partial item images are removed from the set of streaming images or the set of streaming item images, thereby generating a modified set of streaming images or a modified set of streaming item images. Each stream image in the modified set of stream images or each stream item image in the modified set of stream item images is marked (S34) by using a label indicating that the depicted item has been sorted into the at least one unsorted portion (4); and The construction of the at least one classifier (S5) further includes training the machine learning model using a stream image from the modified set of stream images or a stream item image from the modified set of stream item images captured at the first sorting device (1) and each associated label.

7. The method according to any one of the preceding claims, in, The partial image captured at the first sorting device (1) is labeled (S3) using partial tags, which is performed by the processing unit (6), wherein the processing unit (6) is also configured to provide the machine learning model with the partial image captured at the first sorting device and the partial tags.

8. The method according to claims 6 and 7, wherein, The steps according to claim 6 are performed by the processing unit (6), wherein the processing unit is further configured to provide the machine learning model with a stream image from a modified set of stream images or a stream item image from a modified set of stream item images captured at the first sorting device (1) and each associated label.

9. The method according to any one of the preceding claims further comprises: The at least one classifier is distributed (S7) from the first sorting device (1) to the second sorting device (10) via a network infrastructure such as a cloud-based server.

10. The method of claim 9, further comprising: Updating (S8) the at least one classifier distributed to the second sorting device (10), wherein updating the at least one classifier distributed to the second sorting device (10) includes: At the first sorting device (1): Repeat the following steps: sort the item stream (S1) into at least one sorted portion (3) and optionally at least one unsorted portion (4); capture (S2) a set of partial images of items in the at least one sorted portion (3); label (S3) the partial images captured at the first sorting device (1) using partial labels; and construct (S4) at least one classifier, including: training the machine learning model using the partial images captured at the first sorting device (1) and the partial labels, thereby providing at least one updated classifier; Distribute the at least one updated classifier to the second sorting device (10); and The at least one updated classifier is activated at the second sorting device (10).

11. The method according to claim 10, wherein, The update (S8) to the at least one classifier distributed to the second sorting device (10) further includes: sending a verification request from the second sorting device (10) to the first sorting device (1), wherein if the verification request fails, the steps according to claim 10 are performed.

12. The method according to any one of the preceding claims, wherein, Training the machine learning model includes: The partial image and associated partial label data captured at the first sorting device (1), and optionally the stream image and associated stream label data captured at the first sorting device (1), are embedded as data points into a spatial map, such as a two-dimensional or three-dimensional map; and The spatial map is divided into units, each unit having a data point density and a last entry date, the last entry date indicating the time when the latest data point was added to the unit.

13. The method according to claims 10 and 12, wherein, Repeating the step of constructing (S4) at least one classifier also includes: For each unit, the data point density and / or the last entry date are checked, and if the data density of the unit is equal to or higher than a predetermined threshold, and / or if the last entry date of the unit is equal to or earlier than a predetermined threshold, the set of data points contained in the unit is modified, such as by removing each data point from the set of data points.

14. The method according to claims 10 and 12 or claim 13, further comprising repeating the step of constructing (S4) at least one classifier: When adding a new data point to the spatial map, check the data point density of the cell to which the data point will be added and / or the last entry date, and: If the data density of a cell is equal to or higher than a predetermined threshold, the new data point is discarded. and / or If the last entry date of a cell is equal to or earlier than a predetermined threshold, then the new data point is added and at least one adjacent data point in the cell is removed.

15. The method according to any one of the preceding claims, wherein, Constructing (S4) at least one classifier includes: constructing a corresponding classifier for each of the at least one sorted portion (3).

16. A computer-readable storage medium storing instructions for implementing the method according to any one of claims 1 to 14.

Images