Sorter devices using machine learning object sorter models, and related methods

Abstract

A sorter device includes an infeed system, a detection system including at least one emitting device and at least one optical sorter, an ejector, and a computing device operably coupled to the detection system and the ejector. The computing device includes instructions thereon that, when executed by at least one processor, cause an image processing system to generate image data, cause a communications interface to display an output image based on the image data, cause the communications interface to receive a user input including labeling data of at least some pixels defining objects of the commodity in the output image to generate training data, and train a machine learning object sorter model based on the training data. Related sorter devices and methods are also disclosed.

Description

TITLESORTER DEVICES USING MACHINE LEARNING OBJECT SORTER MODELS, AND RELATED METHODSCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of the filing date of EP Application No. 23425047.0, filed September 11, 2023, the disclosure of which is hereby incorporated herein in its entirety by this reference.FIELD

[0002] Embodiments of the present disclosure relate generally to optical bulk sorter devices. More particularly, embodiments of the present disclosure relate to methods of training sorter devices to sort a bulk product, and related sorter devices using machine learning object sorter models and methods.BACKGROUND

[0003] Automatic sorter devices are conventionally utilized to sort bulk products. In particular, sorter devices are typically utilized to separate specific pieces (e.g., grains) from a bulk product in order to be sorted and / or discarded. Sorter devices generally include a conveyor system to create a stream of the bulk product, which may include nuts, grain, seeds, or plastic pieces. The sorter device typically includes optical detection systems arranged for acquiring and analyzing images of the stream of the bulk product. The detection systems are typically configured to provide the image data to a controller, which, based on information determined from the acquired images, sends control signals to an expulsion device to remove selected pieces (e.g., grains) from the bulk product, such as with air jets produced by air nozzles.BRIEF SUMMARY

[0004] According to embodiments of the disclosure, a sorter device includes an infeed system configured to provide a stream of a commodity proximate at least one detection system including at least one emitting device configured to emit electromagnetic radiation and at least one optical sensor configured to measure electromagnetic radiation reflected from the stream,an ejector configured to selectively expose the stream to air to sort the commodity, and a computing device operably coupled to the detection system and the ejector. The computing device includes at least one processor, and at least one non-transitory computer-readable storage medium storing instructions thereon that, when executed by at least one processor, cause the computing device to cause an image processing system to generate image data of the stream based on the measured electromagnetic radiation, cause a communications interface to display an output image based on the image data, and cause the communications interface to receive a user input including labeling data of at least some pixels defining objects of the commodity in the output image to generate training data, and train a machine learning object sorter model based on the training data.

[0005] In some embodiments, the computing device includes instructions thereon that, when executed by the at least one processor, cause the ejector to sort the commodity with the machine learning object sorter model. The instructions may cause the ejector to selectively expose objects of the commodity to air based on the machine learning object sorter model.

[0006] The instructions may, when executed by the at least one processor, cause the image processing system to perform a background thresholding operation on the image data to generate modified image data.

[0007] In some embodiments, when executed by the at least one processor, the instructions may cause the image processing system to perform one or more of a background thresholding operation, a contouring operation, an erosion operation, a growing operation, an area thresholding operation, and a feature extraction operation on the image data to generate modified image data. In some embodiments, the instructions cause the communications interface to display the output image based on the modified image data.

[0008] In some embodiments, the training data may be generated based on a color of the commodity.

[0009] The sorter device may further include a machine learning object sorter system configured to receive additional image data during a sorting operation, and cause an ejector controller to control the ejector based on the additional image data and the machine learningobject sorter model. The machine learning object sorter system may update the machine learning object sorter model based on additional user input.

[0010] The computing device may include instructions thereon that, when executed by at least one processor, cause a machine learning object sorter training system to train the machine learning object sorter model based on only a color of the at least some pixels defining objects of the commodity in the output image.

[0011] In some embodiments, a sorter device includes an infeed system configured to provide a stream of a commodity, at least one emitting device configured to emit electromagnetic radiation at the stream, at least one optical sensor configured to measure electromagnetic radiation reflected from the stream, an ejector configured to selectively expose the stream to air to sort the commodity, and a computing device operably coupled to the detection system and the ejector. The computing device includes at least one processor, and at least one non-transitory computer-readable storage medium storing instructions thereon that, when executed by at least one processor, cause the computing device to cause an image processing system to generate image data of the stream based on the measured electromagnetic radiation, apply a machine learning object sorter model to the image data to generate a labeling decision for objects of the commodity in the stream, and control the ejector based on the labeling decision.

[0012] The computing device may include instructions thereon that, when executed by the at least one processor, cause the computing device to generate the labeling decision based on a color of objects in the image data. In some embodiments, the instructions, when executed by the at least one processor, cause the computing device to modify the image data using one or more of a thresholding operation, an area thresholding operation, and a contouring operation. In some embodiments, the sorter device further includes a machine learning object sorter training system configured to generate the machine learning object sorter model based on user input received by a communications interface of the computing device.

[0013] In some embodiments, a method of operating a sorter device includes obtaining image data of a commodity within a sorter device, displaying, on a communications interface, an output image based on the image data, receiving, on the communications interface, a user inputfor each of a plurality of selected object in the output image, the user input indicating a label for at least some pixels of each of the plurality of selected objects, creating training data from the labels for the at least some pixels of each of the plurality of selected objects, training a machine learning object sorter model to generate a sorting decision for a given object based on an input image of the given object, wherein the machine learning object sorter model is trained based on the labels and associated image data of the plurality of selected objects within the training data, and sorting the commodity with the trained machine learning sorter model.

[0014] The method may further include performing one or more image processing operations on the image data prior to displaying the output image on the communications interface. The modified image may be displayed on the communications interface prior to receiving the user input.

[0015] In some embodiments, the method further includes performing one or more image processing operations on the image data includes performing one or more of a background thresholding operation, a contouring operation, an erosion operation, a growing operation, an area thresholding operation, and a feature extraction operation on the image data to create modified image data. Displaying the output image may include displaying the output image based on the modified image data.

[0016] Sorting the commodity with the machine learning sorter model based on the training set may include selectively controlling operation of an ejector based on sorting decisions generated by the trained machine learning sorter model while continuously receiving additional image data.

[0017] In some embodiments, sorting the commodity with the machine learning sorter model includes receiving additional image data of a stream of the commodity within the sorter device, generating a labeling decision for objects of the commodity within the stream based on the additional image data using the machine learning object sorter model, and providing the labeling decision to an ejector controller.BRIEF DESCRIPTION OF THE DRAWINGS

[0018] While the specification concludes with claims particularly pointing out and distinctly claiming what are regarded as embodiments of the present disclosure, various featuresand advantages may be more readily ascertained from the following description of example embodiments when read in conjunction with the accompanying drawings, in which:

[0019] FIG. 1 is a simplified schematic representation of a sorter device, according to one or more embodiments of the disclosure;

[0020] FIG. 2A is a simplified schematic illustrating a machine learning object sorter training system, in accordance with embodiments of the disclosure;

[0021] FIG. 2B is an example graphical user interface of the sorter device, in accordance with embodiments of the disclosure;

[0022] FIG. 3 is a simplified schematic illustrating a machine learning object sorter system, in accordance with embodiments of the disclosure;

[0023] FIG. 4 is a simplified flow chart illustrating a method of performing a training operation to create a trained machine learning object sorter model, in accordance with embodiments of the disclosure;

[0024] FIG. 5 is a simplified flow chart illustrating a method of sorting a bulk product using a trained machine learning object sorter model, in accordance with one or more embodiments of the disclosure; and

[0025] FIG. 6 is a schematic of a computer-readable storage medium including processor-executable instructions configured to embody one or more of the methods of operating the sorter device.DETAILED DESCRIPTION

[0026] The illustrations presented herein are not actual views of any agricultural machine or portion thereof, but are merely idealized representations to describe example embodiments of the present disclosure. Additionally, elements common between figures may retain the same numerical designation.

[0027] The following description provides specific details of embodiments. However, a person of ordinary skill in the art will understand that the embodiments of the disclosure may be practiced without employing many such specific details. Indeed, the embodiments of the disclosure may be practiced in conjunction with conventional techniques employed in the industry. In addition, the description provided below does not include all elements to form acomplete structure or assembly. Only those process acts and structures necessary to understand the embodiments of the disclosure are described in detail below. Additional conventional acts and structures may be used. The drawings accompanying the application are for illustrative purposes only, and are thus not drawn to scale.

[0028] As used herein, the terms "comprising," "including," "containing," "characterized by," and grammatical equivalents thereof are inclusive or open-ended terms that do not exclude additional, unrecited elements or method steps, but also include the more restrictive terms "consisting of" and "consisting essentially of" and grammatical equivalents thereof.

[0029] As used herein, the term "may" with respect to a material, structure, feature, or method act indicates that such is contemplated for use in implementation of an embodiment of the disclosure, and such term is used in preference to the more restrictive term "is" so as to avoid any implication that other, compatible materials, structures, features, and methods usable in combination therewith should or must be excluded.

[0030] As used herein, the term "configured" refers to a size, shape, material composition, and arrangement of one or more of at least one structure and at least one apparatus facilitating operation of one or more of the structure and the apparatus in a predetermined way.

[0031] As used herein, the singular forms following "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0032] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0033] As used herein, spatially relative terms, such as "beneath," "below," "lower," "bottom," "above," "upper," "top," "front," "rear," "left," "right," and the like, may be used for ease of description to describe one element's or feature's relationship to another element(s) or feature(s) as illustrated in the figures. Unless otherwise specified, the spatially relative terms are intended to encompass different orientations of the materials in addition to the orientation depicted in the figures.

[0034] As used herein, the term "substantially" in reference to a given parameter, property, or condition means and includes to a degree that one of ordinary skill in the art would understand that the given parameter, property, or condition is met with a degree of variance, such as within acceptable manufacturing tolerances. By way of example, depending on the particular parameter, property, or condition that is substantially met, the parameter, property, or condition may be at least 90.0% met, at least 95.0% met, at least 99.0% met, or even at least 99.9% met.

[0035] As used herein, the term "about" used in reference to a given parameter is inclusive of the stated value and has the meaning dictated by the context (e.g., it includes the degree of error associated with measurement of the given parameter).

[0036] As used throughout, ranges are used as shorthand for describing each and every value that is within the range. Any value within the range can be selected as the terminus of the range.

[0037] As used herein, the term "spectral band" means and includes a range of wavelengths of the electromagnetic spectrum. Spectral bands may overlap with one another and / or spectral bands may include non-overlapping ranges of wavelengths. Non-limiting examples of spectral bands include infrared spectral bands (e.g., with wavelengths ranging from about 0.7 pm to about 1.0 mm), the far-infrared region (e.g., with wavelengths ranging from about 10 pm to about 1.0 mm), the mid-infrared region (e.g., with wavelengths ranging from about 2.5 pm to about 10 pm), the near-infrared region (e.g., with wavelengths ranging from about 700 nm to about 2.5 pm), short-wave infrared (IR) region (e.g., with wavelengths ranging from 900 nm to about 1.7 pm, or from about 700 nm to about 2.5 pm), the ultraviolet region (e.g., with wavelengths ranging from about 10 nm to about 400 nm). When referring to a "spectral band" herein, the spectral band is made with reference to a band of electromagnetic radiation having a wavelength falling within a particular range of wavelengths.

[0038] According to embodiments described herein, a sorter device is configured to sort a bulk product (e.g., grain, nuts, seeds, rice, plastic) by selectively removing undesired components from the bulk product. The sorter device includes at least one radiation emitting device (also referred to as an "emitter" or an "emitting device") configured to expose a streamof the bulk product to electromagnetic radiation, and at least one optical sensor configured to measure electromagnetic radiation reflected from and / or transmitted through the stream while exposing the stream to the electromagnetic radiation from the at least one emitting device. The sorter device further includes an ejector configured to selectively introduce pulses of air (e.g., air jets) to the stream to remove undesired components from the stream to sort the bulk product.

[0039] The sorter device further includes a computing device in operable communication with the ejector and with an image processing system. The image processing system is configured to receive image data from the at least one optical sensor (e.g., an intensity of electromagnetic radiation measured by the at least one optical sensor). The image data may correspond to electromagnetic radiation received by the optical sensor during an exposure of the stream to one or more particular spectral bands.

[0040] The computing device may include a machine learning (ML) object sorter model training system configured to train a ML object sorter model, and a ML object sorter system configured to implement the trained ML object sorter model to facilitate sorting of the bulk product. The ML object sorter training system may implement the image processor to receive the image data received by the optical sensor(s), perform one or more operations on the image data to generate modified image data, and display an output image on a communications interface based on the modified image data. The ML object sorter training system may cause the communications interface to receive a user input indicative of a label (e.g., classification) of objects (which may be identified as groups of pixels having a similar color (e.g., measured intensity)) and / or of one or more pixels of an object (e.g., only a defect area of an object) of the bulk product (e.g., individual components of the bulk product) in the output image to assign a label to each object and / or pixel(s) selected by the user. Responsive to receiving the user input and labeling the objects and / or pixels(s), the labeled objects and / or pixels(s) may be provided to a ML model as training data (e.g., ground truth data) and the ML object sorter training system may cause the ML model to be trained to create the trained ML object sorter model based on the user input. Once trained, the ML object sorter model may be configured to identify one or more color features in the image data and a correlation between the one or more color features and the assigned label. Accordingly, the ML object sorter model may be trained to generate asorting decision based on one or more color features of the objects (including only parts of the objects, such as defect areas) in the bulk product, the objects defined within the image data.

[0041] During a sorting operation, the ML object sorter system may cause the image processor to receive image data received by the optical sensor(s) and, optionally, perform one or more operations on the image data. The ML object sorter system may cause the ML object sorter model to receive the image data and classify objects (e.g., groups of pixels) in the image data as either a pass (e.g., do not separate from the bulk product) or a fail (e.g., separate from the bulk product). The classification may be based on an entirety of the object(s), may be based on only defect areas of the objects(s), or both. Responsive to analyzing the image data, the ML object sorter model may generate a labeling decision for each object and an ejector controller may receive the labeling decision. The ejector may receive a sorting decision from the ejector controller to control operations of the ejector and facilitate separation of undesired components from the bulk product.

[0042] Training the ML object sorter model based on the user input defining the labels of the objects including both objects in their entirety and also based on only pixels defining defect areas of the objects may facilitate training the ML object sorter model with improved and more accurate training data and, therefore, improved separation of the bulk product using the ML object sorter model. By way of comparison, conventional object sorter devices may sort based on a feature of an entirety of each object and may not, therefore, consider defect areas of the objects. For example, since some failing objects that should be separated from the bulk product only include relatively small (compared to the overall size of the object) defect areas, conventional object sorter devices may not be configured to perform separation based on color of such objects and may not separate such objects from the bulk product. In addition, training with the user input may facilitate on-site training of the ML object sorter model based on the actual bulk product to be sorted. By way of comparison, conventional sorter devices may include a sorting algorithm based on the particular characteristics of the particular bulk product to be sorted. For example, rice may be separated based on a different sorting algorithm than almonds. In addition, conventional sorting operations may require analyzing the image data in real time, which may limit the speed of the sorting operation. In addition, since the training data is basedon objects in an output image generated by the modified image data, the training data may exhibit less noise and have fewer outliers compared to training data formed from the raw image data. For example, the training data may be based on raw image data that has been passed through one or more of a thresholding operation, a contouring operation, an erosion operation, a growing operation, and an area thresholding operation to remove noise from the raw image data.

[0043] Since the ML object sorter model is configured to classify the objects in the image data (e.g., the bulk product) based on one or more color features, the input to the ML object sorter model includes raw (e.g., unprocessed) image data, which may increase the relative speed of the analysis of the image data compared to conventional sorting operations. For example, the image data may include an abundance of information, which may be difficult to process (e.g., require significant computing resources, computing speed, and time). Using the ML object sorter model may reduce the computing resources and increase the processing speed at which sorting decisions may be made. In addition, the ML object sorter model may facilitate sorting the commodity based only on the color and, therefore, does not require performing object detection on the image data, further improving the processing speed of the sorting decision. Further, the ML object sorter model training system may be configured to create the ML object sorter model for different commodities, without requiring a separate sorting algorithm preloaded on the sorter device.

[0044] FIG. 1 shows a sorter device 100, according to one or more embodiments of the disclosure. The sorter device 100 may include a support frame 102 supporting an infeed system 104, at least one detection system 106, at least one ejector 108, and a plurality of collection bins 110. Furthermore, the sorter device 100 may include a computing device 112 (also referred to as a "system controller") to which the infeed system 104, the at least one detection system 106, and the at least one ejector 108 are operably coupled. The computing device 112 includes an image processing system 114, an ejector controller 115 operably coupled to the image processing system 114, a machine learning (ML) object sorter model training system 117, and a machine learning (ML) object sorter system 121.

[0045] The sorter device 100 may be utilized to sort bulk product (e.g., granular product) 118 such as, for example, nuts, seeds, grain, plastic pieces, etc. The sorter device 100 may sort the pieces of the bulk product 118 (referred to hereinafter as "grains") based on one or more of the color and type of materials of the grains. In particular, the sorter device 100 may sort the grains of the bulk product 118 according to preselected sorting criteria. As a non-limiting example, the sorter device 100 may be utilized to sort grain based on the quality of the grain, which, in some instances, may be determined by a color of the grain. As another non-limiting example, the sorter device 100 may be utilized to sort bulk plastic pieces based on plastic type, which may be determined based on the color of the plastic.

[0046] The infeed system 104 of the sorter device 100 may include a hopper 119 and a chute and / or belt 120. The hopper 119 may define a pathway to the chute and / or belt 120, and in some embodiments, the hopper 119 may include one or more vibrators (e.g., a vibrator feeder), augers, or other feeders to feed the bulk product 118 from the hopper 119 to the chute and / or belt 120. The chute and / or belt 120 may be sized, shaped, and oriented to cause the bulk product 118 to descend due to gravity and / or a conveyor belt to pass in front of the at least one detection system 106. For instance, the chute and / or belt 120 may be configured to produce a stream 122 of the bulk product 118 to pass in front of the at least one detection system 106 according to a selected velocity (e.g., speed).

[0047] The at least one detection system 106 may include a plurality of emitting devices 124, one or more background elements 126, one or more optical sensors 128, a plurality of intensity sensors 130, and a plurality of reference elements 132. The plurality of emitting devices 124 may be configured to emit electromagnetic radiation at the stream 122 of the bulk product 118 as the stream 122 passes in front of the at least one detection system 106. For example, the plurality of emitting devices 124 may include one or more light-emitting-diodes (LEDs) for emitting light. In some embodiments, the plurality of emitting devices 124 may emit one or more of visible light, short-wave infrared light (SWIR light), near infrared light (NIR light), infrared (IR) light, or ultra-violet (UV) light. In one or more embodiments, the plurality of emitting devices 124 are configured to emit light within a specific (e.g., selected) spectral band of the electromagnetic spectrum. In some embodiments, the plurality of emitting devices 124 includesat least four emitting devices. In one or more embodiments, at least one of the at least four emitting devices 124 is configured to emit a first type of electromagnetic radiation (e.g., UV light), and at least one of the at least four emitting devices is configured to emit a second type of electromagnetic radiation (e.g., NIR light). In yet other embodiments, each of the emitting devices 124 is configured to emit electromagnetic radiation within a different spectral band within the infrared (IR) region.

[0048] By way of non-limiting example, the emitting devices 124 may individually be configured to emit electromagnetic radiation within a particular spectral band within the infrared (IR) region (such as within the IR-A band having wavelengths within the range of from about 1.4 pm to about 3.0 pm). In some embodiments, one or more of the emitting devices 124 is configured to emit electromagnetic radiation within the short-wave IR (SWIR) spectrum, such as within the range of from about 900 nm to about 2.5 pm.

[0049] The one or more optical sensors 128 may include one or more of a charged- coupled device (CCD) camera, an IR camera, a UV camera, or an RGB camera. During use and operation, the one or more optical sensors 128 may be oriented and configured to detect (e.g., capture) electromagnetic radiation reflected from the stream 122 and / or electromagnetic radiation transmitted through the stream 122 of the bulk product 118 due to the plurality of emitting devices 124. For instance, the field of view of the one or more optical sensors 128 may include at least a portion of the stream 122 of bulk product 118. In one or more embodiments, the at least one detection system 106 may further include one or more optical filters for filtering (e.g., narrowing) the reflected light being detected (e.g., captured) by the one or more optical sensors 128. In some embodiments, the one or more optical filters may narrow the reflected light into specific (e.g., selected) wavelengths that may accentuate sorting criteria (e.g., criteria distinguishing grades or types of bulk products).

[0050] As a non-limiting example, in some embodiments, the sorter device 100 includes a monochromatic sorter device and the at least one detection system 106 includes a single optical filter between the stream 122 of the bulk product 118 and a respective optical sensor 128. The single optical filter may produce, for example, a light / dark separation. As another non-limiting example, in some embodiments, the sorter device 100 includes a bi-chromatic sorter device, theat least one detection system 106 includes two optical filters between the stream 122 of bulk product 118 and a respective optical sensor 128. For example, the at least one detection system 106 may include any two conventional optical filters.

[0051] As noted above, the at least one detection system 106 may include one or more background elements 126, and the one or more background elements 126 may be disposed and oriented behind the stream 122 relative to the one or more optical sensors 128. The background elements 126 may facilitate better detection and imaging of the individual grains of the bulk product 118 compared to sorter devices 100 not including the background elements 126. The one or more background elements 126 may include any known background elements used in sorter devices.

[0052] The plurality of intensity sensors 130 may be oriented relative to the plurality of emitting devices 124 such that the plurality of intensity sensors 130 may be utilized to measure an intensity of the electromagnetic radiation emitted by the plurality of emitting devices 124. Furthermore, and as noted above, the at least one detection system 106 may include a plurality of reference elements 132. Each reference element 132 may be disposed within a field of view of at least one optical sensor 128. The intensity sensors 130 and the reference elements 132 may individually be substantially similar to (e.g., the same as) the respective intensity sensors and reference sensors described in European Patent Application No. EP23425017.3, "[]Sorter Devices, Detection [] Systems of Sorter Devices, and Related Methods," filed April 28, 2023. In addition, the sorter device may include a plurality of current sensors, as described in European Patent Application No. EP23425017.3.

[0053] As is described briefly above, each of the infeed system 104, the at least one detection system 106, and the at least one ejector 108 may be operably coupled to and at least partially operated by the computing device 112. For example, the computing device 112 may be configured to provide control signals to the infeed system 104 to cause the infeed system 104 to feed the bulk product 118 to the chute and / or belt 120 and to generate the stream 122. Furthermore, the computing device 112 may be configured to provide control signals to the plurality of emitting devices 124 and the optical sensors 128 to control operation of the plurality of emitting devices 124 and the optical sensors 128. The computing device 112 may include theimage processing system 114 that may receive image data (e.g., sets of image data) from the plurality of optical sensors 128, analyze the image data, and generate output image data that is received by one or both of the ML object sorter model training system 117 and the ML object sorter system 121 (e.g., depending on whether the sorter device 100 is performing a training operation or a sorting operation). For example, during a training operation, the computing device 112 may cause the ML object sorter model training system 117 to receive the image data from the image processing system 114 to generate training data (e.g., a training set of data) and train a ML object sorter model based on the training data. During a sorting operation (e.g., after a training operation has been completed and the computing device 112 includes a trained ML object sorter model), the computing device 112 may cause the ML object sorter system 121 to receive image data from the image processing system 114 and to classify objects in the image data (corresponding to objects of the bulk product 118 in the stream 122) based on one or more color properties of the objects.

[0054] During a sorting operation (e.g., after a training operation has been completed and the computing device 112 includes a trained machine learning object sorter system 121), the computing device 112 may cause the ML object sorter system 121 to receive image data from the image processing system 114 and to generate label decisions for each object in the image data using the ML object sorter model. The label decisions may be received by the ejector controller 115 and cause the ejector controller 115 to cause the at least one ejector 108 to operate (e.g., selectively open one or more nozzles thereof to selectively remove one or more pieces of the bulk product 118 as the bulk product 118 passes by the respective one or more nozzles) based on the label decisions.

[0055] FIG. 2A is a simplified schematic illustrating the ML object sorter model training system 117 and generating a ML object sorter model 252 (also referred to as a "trained ML object sorter model 252"), in accordance with embodiments of the disclosure. The system 200 includes the image processing system 114 configured to receive input image data 202 from the optical sensors 128. In some embodiments, the image data 202 includes video data including two or more frames of image data. The image data 202 may include data received from the optical sensors 128 during operation of the sorter device 100. In some embodiments, the image data202 includes an intensity of electromagnetic radiation measured (e.g., reflected and / or transmitted through the stream 122) with the optical sensors 128 during exposure of the stream 122 of bulk product 118 to electromagnetic radiation from the emitting device(s) 124. The image data 202 may be in the form of pixels, each pixel indicative of an intensity of measured electromagnetic radiation and corresponding to a particular location within the sorter device 100.

[0056] The image data 202 may include single channel image data or may include multichannel image data. In some embodiments, the image data 202 includes multilayer image data, each layer including single channel image data indicative of a different color (e.g., RGB). By way of non-limiting example, the image data 202 may include RGB image data.

[0057] The image processing system 114 may be configured to analyze the image data 202 and generate modified image data 206. The image processing system 114 may include one or more computer vision operation 204 for modifying the image data 202 (e.g., preprocessing the image data 202) to generate the modified image data 206. For example, the one or more of computer vision operations 204 may include a thresholding operation 208, a contouring operation 210, an erosion operation 212, a growing operation 214, an area thresholding operation 216, and a feature extraction operation 217.

[0058] The thresholding operation 208 be configured to perform a thresholding operation on the image data 202. The thresholding operation may be performed on single channel image data (e.g., on individual channels of the image data 202) or may be performed on multichannel image data. In some embodiments, the thresholding operation is performed on RGB image data, such as on each component of RGB image data. The thresholding operation may include a segmentation technique configured to facilitate separating groups of pixels defining an object of the bulk product 118 from the background. The thresholding operation may be based on one or more of a histogram-based thresholding method, a clustering-based thresholding method, an entropy-based thresholding method, an object attribute-based thresholding method, and a spatial thresholding method. In some embodiments, the thresholding operation includes a background thresholding operation configured to exclude the background image data from the image data 202. The groups of pixels may share one or more color characteristics. For example, the groups of pixels may be defined by an intensity value within a predetermined range(corresponding to a predetermined range of color shades), each pixel within the predetermined range neighbored by at least another pixel defined by an intensity within the predetermined range.

[0059] The contouring operation 210 may include a contour detection operation configured to identify edges of groups of pixels defining objects of the bulk product 118 in the image data 202, and may also be referred to as an edge detection operation. The contouring operation may identify pixels in the image data 202 corresponding to (e.g., defining) a contour of the groups of pixels in the image data 202. In some embodiments, responsive to performing the contour operation, the contouring operation 210 may be configured to selectively remove the pixels defining the contour of each group of pixels defining object of the bulk product 118 from the image data 202.

[0060] The erosion operation 212 may be configured to perform an erode operation to erode boundaries of regions of foreground pixels, which may shrink a size (a number of pixels) of foreground pixels in the image data 202 and increase the size of holes within those areas. In some embodiments, the structuring element (also referred to as the "kernel") of the erosion operation 212 is a square with 9 pixels (e.g., a 3x3 square; 8 pixels surrounding an input pixel (e.g., anchor point or anchor pixel)) or a cross with 5 pixels (e.g., a 4 pixels; one above, one below, one to the right, and one to the left of the input pixel).

[0061] The growing operation 214 may include a growing operation (also referred to as a dilation operation) configured to replace an input pixel with a maximum pixel value of pixels in a kernel as the kernel is scanned across the image data, which may cause bright regions in the image data to grow. In some embodiments, one or both of the erosion operation 212 and the growing operation 214 may be applied to the image data 202 to remove noise in the image data 202 and / or to isolate individual groups of pixels (e.g., corresponding to objects) of the bulk product 118 in the image data 202. By way of non-limiting example, in some embodiments, the image processing system 114 performs an opening operation (an erosion operation followed by a growing operation) to generate the modified image data 206. In some embodiments, the image processing system 114 performs a closing operation (e.g., a growing operation followed by anerosion operation), in some embodiments, the combination of the erosion operation and the growing operation removes defect areas from the image data 202.

[0062] The area thresholding operation 216 may be configured to perform an area thresholding operation on the image data 202 to remove groups of pixels in the image data 202 defined by less than a predetermined number of pixels and / or to remove groups of pixels from the image data 202 defined by more than a different predetermined number of pixels.

[0063] The feature extraction operation 217 may be configured to process the image data 202 to enhance differences in one or more features (e.g., colors, the measured intensity) of the pixels in the image data 202. In some embodiments, the feature extraction operation 217 is configured to determine a correlation between the one or more features (e.g., colors, measured intensity) of the pixels in the image data 202. In some embodiments, the feature extraction operation 217 includes a principal component analysis (PCA) algorithm.

[0064] With continued reference to FIG. 2A, the image processing system 114 may cause a communications interface (e.g., a display, a user interface) 218 of the computing device 112 to receive the modified image data 206 and to display an output image 220 based on the modified image data 206 on a display of the communications interface 218. The output image 220 may be a visual representation of the modified image data 206. The output image 220 may be displayed as a graphical user interface (GUI) 222. The GUI 222 may include the output image 220 and may further include a menu 224 through which the user may select one or more options configured for facilitating operation of the sorter device 100.

[0065] FIG. 2B is an example of the GUI 222 that is displayed by the communications interface 218 (FIG. 2A). With collective reference to FIG. 2A and FIG. 2B, the GUI 222 includes the output image 220. In addition, the GUI 222 includes the menu 224. The communications interface 218 may be configured to receive user input 226 via the GUI 222, such as via the menu 224. In some embodiments, the menu 224 includes multiple toolbars, such as a labeling toolbar 227, a modification toolbar 228, a filtering toolbar 230, and an additional toolbar 232.

[0066] The labeling toolbar 227 may include selectors (e.g., buttons, toggles) for indicating a status of one or more objects within the output image 220. For example, the labeling toolbar 227 may include a first selector 234 configured to facilitate the communications interface218 receiving a user input 226 indicative of a label of the objects (e.g., one of which is illustrated in box 238) and / or one or more pixels within the output image 220. For example, the first selector 234 may facilitate labeling an object and / or one or more pixels as "passing" (e.g., "good") indicating that the objects and / or pixels should not be separated from the bulk product 118 or labeling the object and / or one or more pixels as "failing" (e.g., "bad"), indicating that the objects and / or pixels should be separated from the bulk product 118. By way of non-limiting example, in use and operation, a user may select "mark good" (e.g., pass) on the first selector 234 and select (e.g., by touching) an object and / or one or more pixels on the GUI 222 to add the object and / or one or more pixels to a passing group; and the user may select "mark bad" (e.g., fail) on the first selector 234 and select (e.g., by touching) an object and / or one or more pixels on the GUI 222 to add the object and / or one or more pixels to a failing group. The labeling toolbar 227 may include a second selector 236 configured to facilitate toggling between labeling an object or one or more pixels (e.g., a group of pixels) when assigning a label to an object and / or pixel(s) using the first selector 234. For example, the second selector 236 may be configured to be toggle between "mark object" and "mark pixel." In some embodiments, responsive to receiving a user input 226 to assign a label using the first selector 234, if the second selector 236 is toggled to "mark object" all of the pixels defining the marked object will be assigned the selected label. In addition, responsive to receiving a user input 226 to assign a label using the first selector 234, if the second selector 236 is toggled to "mark pixel" only the selected pixels, which may be less than all of the pixels defining an entire object, will be assigned the selected label. The GUI 222 may include one or more selectors in the labeling toolbar 227 configured to facilitate selection of the desired pixels to assign the label.

[0067] As described in further detail herein, the user may select multiple objects and / or multiple pixel(s) in the output image 220 and classify each object and / or pixel(s) as either passing or failing (good or bad) based on visual inspection by the user. The user may provide the user input 226 to the communications interface 218 via the GUI 222 to indicate which objects and / or pixel(s) pass and would not be separated from the bulk product 118 and which objects and / or pixel(s) fail and would be separated from the bulk product 118. In some embodiments, the user may determine a quality of the objects and / or pixel(s) of the commodity based on one or morefeatures of the object and / or pixels(s), such as one or more of a quality, color, a size, a contour, a shape, and a chemical characteristic.

[0068] With continued reference to FIG. 2B, the modification toolbar 228 may include a defect toolbar 240 and a pass toolbar 242. In some embodiments, responsive to selection of an object in the output image 220 and classification of the selected object and / or pixel(s) as a fail (such as with the first selector 234) the communications interface 218 may be configured to receive instructions to perform a grow operation and / or an erode operation on pixels and may further be configured to identify the color(s) of the object and / or pixel(s). Similarly, responsive to selection of an object and / or pixels(s) in the output image 220 and classification thereof as passing (such as with the second selector 236), the communications interface 218 may be configured to perform a grow operation and / or an erode operation on the pixe l(s)s (e.g., of the object as a whole or on only the selected pixel(s)) and may further be configured to define the color(s) thereof. In some embodiments, the modification toolbar 228 facilitates receipt of user input 226 to cause the image processing system 114 to perform one or more image processing techniques on the image data and generate modified image data, as described in additional detail herein.

[0069] Accordingly, a user may label objects and / or pixel(s) within the output image 220 that pass and / or objects and / or pixel(s) that fail via the GUI 222, such as with one or more of the first selector 234, the second selector 236, the defect toolbar 240, and the pass toolbar 242. As described in further detail herein, the ML object sorter model training system 117 may be configured to train a ML object sorter model to classify objects and / or pixel(s) within image data (e.g., different output images obtained during operation of the sorter device 100) based on the user input 226. By way of non-limiting example, in some embodiments, the user input 226 may label each object (e.g., pass or fail) and the ML object sorter model training system 117 may use the labeled objects as training data to train the ML object sorter model. In addition, the user input 226 may label only some pixels of objects (e.g., pass or fail), such as only defect areas of the objects, and the ML object sorter model training system 117 may use only labeled pixels as training data to train the ML object sorter model. In some embodiments, the boundaries of theobject may be defined using the defect toolbar 240 and / or the pass toolbar 242, as described in further detail herein.

[0070] The filtering toolbar 230 may include one or more selectors configured for performing one or more operations on the output image 220, such as for applying one or more filters on the output image 220. The additional toolbar 232 may include one or more selectors configured to facilitate operation of the selector device 100.

[0071] Referring back to FIG. 2A, responsive to receiving the user input 226, the system 200 is configured to generate training data (e.g., a training set of data) 244 based on the user input 226. The training data 244 may include labels including image data individually including an object and metadata including a label of the object in the image data. In other words, the training data 244 may include labeled objects and / or labeled pixel(s) based on the user input 226. The training data 244 may include, for example, a pass label 246 (also referred to as a "good label") including objects and / or pixel(s) labeled as passing, and a fail label 248 (also referred to as a "bad label") including objects and / or pixel(s) labeled as failing. The training set of data 244 may also be referred to herein as "ground truth data" and may be used for training a ML object sorter model. In addition to the labels and other characteristic data associated with the corresponding objects, in one or more embodiments, the training data 244 additionally includes the content of the corresponding objects, such as image data or other characteristics derived from the image data generated and / or presented to the user.

[0072] The system 200 further includes a machine learning system including ML object sorter model 252 generated at least in part, based on the training data 244. In some such embodiments, the ML object sorter model 252 includes a supervised ML model. The ML object sorter model 252 may include a classification ML model configured to classify (e.g., label) an object and / or pixel(s) based on one or more features thereof. The ML object sorter model 252 may include one or more of a decision tree algorithm, a random forest algorithm, a Gaussian mixture model (GMM), a logistic regression algorithm (e.g., linear regression, lasso regression, ridge regression, polynomial regression), a naive bayes algorithm, a K-nearest neighbors (KNN) algorithm, a support vector machine (SVM) algorithm, a stochastic gradient descent (SGD) algorithm, and a half of threshold (sHoT) training algorithm. In some embodiments, the ML objectsorter model 252 includes one or more of a decision tree model, a random forest model, and a Gaussian mixture model.

[0073] In some embodiments, the ML object sorter model 252 receives the training data 244 including the pass labels 246 and the fail labels 248 and the associated image data and train the ML object sorter model 252The ML object sorter model 252 may be configured to facilitate classification and sorting of multiple types of commodities (bulk products) based (e.g., based only) on image data (e.g., the color of pixels in the image data). By way of non-limiting example, the ML object sorting model 252 may include multiple classification algorithms 254, such as a classification algorithm 254 for each type of bulk product that may be sorted by the sorter device 100. By way of non-limiting example, the ML model 252 may include a classification algorithm for one or more of grain (e.g., each of wheat, barley, oats, corn, rye, quinoa, bulgar), nuts (e.g., each of peanuts, almonds, cashews, etc.), seeds, rice, and plastic. In some such embodiments, the ML object sorter model 252 may be trained with different sets of training data244 specific to each of the classification algorithms 254 (e.g., the ML object sorting model 252 for classifying rice may be trained with image data 202 obtained while the stream 122 includes rice).

[0074] In some embodiments, the ML object sorter model 252 includes determining a correlation between one or more color features of the objects (e.g., the pixels defining the objects) and / or pixel(s) in the training data 244. The color features may include an average intensity of measured radiation in one or more (e.g., each) channel of image data defining a group of pixels, a number of pixels in a group of pixels having an intensity outside of a predetermined range (e.g., corresponding to pixels in the groups of pixels having an undesired color), or another color feature. In some embodiments, the ML object sorter model 252 is trained to predict a class of objects and / or pixel(s) (e.g., pass or fail) based on pixel data including the measured electromagnetic radiation and corresponding to one or more color features of the objects and / or pixel(s) in the image data.

[0075] With continued reference to FIG. 2A, in some embodiments, a portion of the training data 244 is provided to the ML object sorter model 252 as test data 256 configured to test the ML object sorter model 252. By way of non-limiting example, the test data 256 includeslabeled data (the label defined by the user input 226). In some embodiments, the test data 256 may be received by the ML object sorter model 252 and the ML object sorter model 252 performs a classification of the test data 256 to determine an accuracy of the ML object sorter model 252. In some such embodiments, the test data 256 may be used to determine one or more of the precision of the ML object sorter model 252 for the particular classification algorithm 254 (e.g., the percentage of correct classifications), the recall of the ML object sorter model 252 (e.g., the percentage of bad labels in the test data 256 identified by the ML object sorter model 252), and the Fl-score (e.g., an average metric accounting for the precision and the recall). The test data 256 may be displayed on the GUI 222 so that the user may determine a suitability of the ML object sorter model 252 prior to use of the ML object sorter model 252 in a sorting operation.

[0076] The ML object sorter model 252 may be continuously or intermittently updated with additional training data. The ML object sorter model 252 may receive incremental learning data (e.g., additional user input including labeled image data of image data obtained during a sorting operation) and / or may receive online machine learning data.

[0077] FIG. 3 is a schematic representation of the ML object sorter system 121, in accordance with embodiments of the disclosure. The ML object sorter system 121 includes image data 302 received from one or more of the optical sensors 128. The image data 302 may be substantially the same as the image data 202, except that the image data 302 may be obtained during a sorting operation rather than during a training operation. For example, the image data 302 may be obtained during sorting of a bulk product 118 after the ML object sorter model 252 has been trained using the ML object sorter model training system 117 (FIG. 2A).

[0078] In some embodiments, the image processing system 114 may receive the image data 302 and analyze the image data to identify one or more regions within the image data 302 corresponding to one or more objects (e.g., one or more objects of the bulk product 118) in the stream 122. By way of non-limiting example, the image processing system 114 may be configured to identify groups 304 of pixels (also referred to as "clusters" of pixels) in the image data 302 (e.g., pixels neighboring one another) having the same color (e.g., substantially the same color) and / or having an intensity value in one or more channels within a predetermined range. In some embodiments, the groups 304 of pixels may include pixels having an intensity within apredetermined range, corresponding to a shade of one or more colors. In some such embodiments, the groups 304 of pixels may be identified based on the detected color of the bulk product 118 in the stream 122. In some embodiments, a number of the groups 304 of pixels may correspond to a number of objects of the bulk product 118 captured in the image data 302. One or more of the groups 304 of pixels may correspond to an object and / or one or more of the groups 304 of pixels may correspond to an area (e.g., a defect area) of an object of the bulk product 118.

[0079] In some embodiments, the image processing system 114 may perform one or more image processing operations on the image data 302, as described above with reference to FIG. 2A. In some embodiments, during the sorting operation, the image processing system 114 performs one or more different operations and / or sequences of operations on the image data 302 compared to the operations performed on the image data 202 during the training operation described above with reference to FIG. 2A. By way of non-limiting example, in some embodiments, the image processing system 114 performs an area thresholding operation 216 on the image data 302 to identify groups 304 of pixels in the image data 302 defined by more than a predetermined number of pixels. The image processing system 114 may further perform one or more of a background thresholding operation 208, a contouring operation 210, an erosion operation 212, a growing operation 214, and a feature extraction operation 217 on the image data 302.

[0080] In other embodiments, the image processing system 114 may be configured to perform one or more object detection techniques to identify the objects of the bulk product 118 within the stream 122 rather than or in addition to identifying the groups 304 of pixels.

[0081] Responsive to generating the groups 304 of image data, the ML object sorter system 121 may cause the ML model 252 to receive each group 304 of image data and analyze each group 304 of image data to generate a labeling decision 306, which may also be referred to herein as a "sorting decision". The ML model 252 may create the labeling decision 306 based on, for example, one or more color features of the pixels in the group 304 of pixels. By way of nonlimiting example, the ML object sorter model 252 may determine an average intensity value ofthe pixels of a group 304 of pixels and compare the average intensity value of the pixels of the group 304 to the training set of data.

[0082] The labeling decision 306 may include, for example, a decision to label the object and / or pixel(s) defined by the group 304 of pixels as passing (e.g., good) or to label the object and / or pixel(s) as failing (e.g., bad). The labeling decision 306 may be made based on the ML object sorter model 252.

[0083] With continued reference to FIG. 3, the labeling decision 306 may be received by the ejector controller 115 and may cause the ejector controller 115 to cause the ejector 108 to operate. For example, responsive to receiving the labeling decision 306, the ejector controller 115 may cause the ejector 108 to direct a stream of air to the object defined by the group 304 of pixels to separate the object from the stream 122 for groups 304 of pixels labeled as failing (or bad). Responsive to receiving a passing labeling decision 306, the ejector controller 115 may cause the ejector 108 not to direct a stream of air at the object.

[0084] In some embodiments, the ML object sorter system 121 may be configured to provide additional training data to the ML object sorter model 252 for further training thereof. By way of non-limiting example, at least some of the groups 304 of pixels may be received by a communications interface 218. The communications interface may display the image data 302 as an output image 320 on a GUI 222, as described above with reference to FIG. 2A. In some embodiments, a predetermined percentage of the groups 304 of pixels may be received by the communications interface 218.

[0085] A user may view the output image 320 and provide user input 326 to the ML object sorter system 121 and / or the ML object sorter model training system 117 via the communications interface 218. The user input 326 may include a label for the output image 320, such as whether the object and / or pixel(s) of the bulk product 118 in the output image 320 should be labeled as a pass or a fail.

[0086] Responsive to receiving the user input 326, the ML object sorter model training system 117 or the ML object sorter system 121 may perform a compare operation 308 to compare the user input 326 to the labeling decision 306 and determine whether the ML object sorter model 252 correctly labeled the image data 302. An output 330 of the compare operation308 may be received by the ML object sorter model 252 and may be configured to facilitate continuous training of the ML object sorter model 252.

[0087] In other embodiments, the ML object sorter model 252 is not continuously trained and the labeling decision 306 is not compared to the user input 326.

[0088] FIG. 4 is a simplified flow diagram illustrating a method 400 of training a ML object sorter model, in accordance with embodiments of the disclosure. The method 400 includes receiving image data with one or more optical sensors 128, as shown in act 402. The image data may include, for example, the image data 202. The image data may be obtained during operation of the sorter device 100, such as du ring flow of a stream 122 of the bulk product 118 in front of the at least one detection system 106.

[0089] The method 400 may further include processing the image data to create modified image data, as shown in act 404. The image data may be processed by performing one or more operations on the image data, such as one or more of a thresholding operation 208, a contouring operation 210, an erosion operation 212, a growing operation 214, an area thresholding operation 216, and a feature extraction operation 217, as described above with reference to FIG. 2A. Creating the modified image data may include sequentially performing a background thresholding operation 208 on the image data, performing a contouring operation 210 on the image data, and performing a feature extraction operation 217 on the image data. In some embodiments, responsive to performing the contouring operation 210, creating the modified image data may include performing one or both of an erosion operation 212 and a growing operation 214 on the image data, followed by performing an area thresholding operation on the image data.

[0090] Responsive to generating the modified image data, the method 400 may further include displaying an output image based on the modified image data, as shown in act 406. Displaying the output image may include displaying an output image on a communications interface 222, such as a display (e.g., a monitor).

[0091] The method 400 may further include receiving a user input indicative of a label of an object and / or pixel(s) in the output image and creating training data, as shown in act 408. In some embodiments, responsive to displaying the output image, a user may select an objectand / or pixel(s) in the output image, such as by touching the object and / or pixel(s) on a display to select the object and / or pixel(s). After viewing the selected object and / or pixel(s) in the output image, the user may label the object and / or pixel(s) (e.g., either as a pass or a fail) based on the appearance thereof. By way of non-limiting example, the user may label the object and / or pixel(s) based on one or more features of thereof, such as one or more of a color, a variation in color, a shape, a size, and a texture thereof.

[0092] The user input may be provided to the communications interface 218 and may be configured to classify the object and / or pixel(s). For example, as described above with reference to FIG. 2A and FIG. 2B, responsive to receiving the user input, the ML object sorter model training system 117 may create training data. Responsive to receiving user input from a plurality of output images, the method 400 may include creating a set of training data.

[0093] Responsive to receiving creating the training data, the method 400 may include training a machine learning model based on the training data to create a trained ML object sorter model 252, as shown in act 410. The ML object sorter model 252 may include a decision tree model, a random forest model, or a Gaussian mixture model. The ML object sorter model training system 117 may train a ML model to create the ML sorting model 252 based on the training data. The ML sorting model 252 may be trained to sort the same type of object from which the training data is generated.

[0094] In some embodiments, the ML sorting model 252 may be trained based on only a color feature of the labeled objects and / or pixel(s) in the training data. In some embodiments, the ML object sorter model 252 may identify patterns in the color the objects and / or pixel(s) with a passing label 246 and objects and / or pixel(s) with a failing label 248 (e.g., the intensity pixels defining the objects, an average intensity of the pixels defining the objects and / or pixel(s), one or more other patterns in the color data of the labeled objects and / or pixel(s)) to create the ML sorting model 252.

[0095] Accordingly, the ML object sorter model 252 may be trained based on user input defining the labels of the objects and / or pixel(s). In addition, because the training data is determined based on the modified image data 206, the input to the ML object sorter model training system 117 may be more accurate and of higher quality than input data including rawimage data (e.g., the image data 202). Training the ML object sorter model 252 with higher quality training data (e.g., the modified image data 206) may facilitate creating the ML object sorter model 252 to exhibit a higher accuracy than a model trained with only the raw image data 202. In addition, by training the ML object sorter model 252 to classify the objects and / or pixel(s) based only on color data (e.g., intensity data) of pixels defining an image, the ML object sorter model 252 may classify (label) objects and / or pixel(s) faster than conventional methods of sorting a bulk product. Accordingly, in some embodiments, since only some pixels of an object (e.g., only regions of an object having defect areas) may be labeled and provided as training data, the ML object sorter model 252 may be trained with better training data than labeling the entire object which may include only small or minor regions having defect areas whereas other regions (e.g., a majority) of the object would be passing (e.g., have passing colors). For example, some objects may include only some areas that include defects. Training the ML object sorter model 252 with fail labels 248 defined only by the defect areas and not the entire object (which may include some pixels that would otherwise be labeled with a pass label 246) facilitates training the ML object sorter model 252 with improved training data and improved performance of the ML object sorter model 252.

[0096] FIG. 5 is a simplified flow diagram illustrating a method 500 of sorting a bulk product using the ML object sorter model 252, in accordance with embodiments of the disclosure. The method 500 includes receiving image data during a sorting operation, as shown in act 502. Receiving the image data may be substantially the same as described above with reference to act 402, except that the image data may be generated during a sorting operation.

[0097] Responsive to receiving the image data, the method 500 may optionally include creating modified image data, as shown in act 504. Creating the modified image data may include performing one or more of a thresholding operation 208, a contouring operation 210, an erosion operation 212, a growing operation 214, an area thresholding operation 216, and a feature extraction operation 217 on the image data, as described above with reference to FIG. 3. By way of non-limiting example, in some embodiments, the image processing system 114 performs an area thresholding operation 216 on the image data to identify groups 304 of pixels in the image data defined by more than a predetermined number of pixels and further performs one or moreof a background thresholding operation 208, a contouring operation 210, an erosion operation 212, and a growing operation 214, as described with reference to FIG. 3. In other embodiments, image processing system 114 may be configured to perform one or more object detection techniques to identify the particles of the bulk product 118 within the stream 122 rather than or in addition to identifying the groups of pixels.

[0098] The method 500 may further include analyzinggroups of pixels of the image data using a ML object sorting model to generate a labeling decision for each group of pixels, as shown in act 506. The ML object sorting model 252 may analyze groups (e.g., groups 304) of image data to identify one or more color features of the groups of image data and generate the labeling decision based on the one or more color features. By way of non-limiting example, the ML sorting model 252 may determine an average intensity value of the pixels of a group of pixels and compare the average intensity value of the pixels of the group 304 to the training set of data.

[0099] The labeling decision may include an identification of whether each object defined by each group of pixels is a pass (e.g., should remain with the bulk product 118) or a fail (e.g., should be separated from the bulk product 118) and may be determined based on the ML object sorting model 252.

[0100] Responsive to generating the labeling decision with the ML object sorter model 252, the method 500 includes receiving the labeling decision with an ejector controller 115, as shown in act 508. For example, the ejector controller 115 may receive the labeling decision from the ML object sorter model 252.

[0101] Responsive to receiving the labeling decision, the method 500 includes controlling an ejector 108 based on the labeling decision, as shown in act 510. In some embodiments, the ejector 108 receives a control signal from the ejector controller 115 responsive to the ejector controller 115 receiving the labeling decision. In some embodiments, responsive to receiving a fail labeling decision for an object from the ML object sorting model 252, the ejector controller 115 may send a signal to the ejector 108 to expose the object to a stream (e.g., pulse) of air to separate the object from the bulk product 118; and responsive to receiving a pass labeling decision for an object from the ML object sorting model 252, the ejector controller 115 may control the ejector 108 to not expose the object to a stream (e.g., pulse) ofair to separate the object from the bulk product 118 (e.g., by not sending a signal to the ejector 108).

[0102] Accordingly, the sorter device 100 may be configured to receive training data from a user and to train a ML object sorting model to label objects to be sorted based on one or more color properties of the objects and / or one or more color properties of one or more regions (e.g., defect areas) of the objects. In addition, the sorter device 100 may be configured to perform a sorting operation of the bulk product 118 based on the ML object sorting model 252 (and without a predefined algorithm for sorting the particular bulk product 118). Training the ML object sorting model 252 to facilitate the sorting operation may provide advantages compared to conventional methods of sorting a bulk product. For example, conventional methods of sorting may sort based on wavelengths of electromagnetic radiation that are reflected from the bulk product. However, the color of the bulk product may depend on the type of bulk product to be sorted. As one example, different types of rice may exhibit different properties color properties. In addition, the sorter device 100 may not require a different algorithm for each type of product to be sorted. Rather, the sorter device 100 may be trained to sort each unique bulk product based on the user input that defines the training data train the ML object sorting model for each unique type of bulk product. Further, the labeling decision (and the sorting of the objects) may be based solely on the image data and the ML object sorting model 252 without modification of the image data, which may be required for some conventional sorting operations.

[0103] While the machine learning object sorter model training system 117 and the ML object sorter system 121 have been described and illustrated as being located on the same computing device 112 of the sorter device 100, the disclosure is not so limited. In other embodiments, the machine learning object sorter model training system 117 may be located remotely from the sorter device 100. In some such embodiments, the user input 226 may be received at the sorter device 100 and communicated to the machine learning object sorter model training system 117 over a network. The machine learning object sorter model training system 117 may train the ML object sorter model 252 (as described above with reference to FIG. 2A and FIG. 4) and the sorter device 100 may receive the trained ML object sorter model 252 for implementation on the sorter device 100, such as on the ML object sorter system 121.

[0104] FIG. 6 is a schematic view of a computer device 602, in accordance with embodiments of the disclosure. In some embodiments, the computing device 112 (FIG. 1) includes a computer device such as the computer device 602 of FIG. 6. The computing device 602 may include a communication interface 604, at least one processor 606, a memory 608, a storage device 610, an input / output device 612, and a bus 614. The computing device 602 may be used to implement various functions, operations, acts, processes, and / or methods disclosed herein, such as the method 400 and / or the method 500.

[0105] The communication interface 604 may include hardware, software, or both. The communication interface 604 may provide one or more interfaces for communication (such as, for example, packet-based communication) between the computing device 602 and one or more other computing devices or networks (e.g., a server). As an example, and not byway of limitation, the communication interface 604 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a Wi-Fi.

[0106] The at least one processor 606 may include hardware for executing instructions, such as those making up a computer program. By way of non-limiting example, to execute instructions, the at least one processor 606 may retrieve (or fetch) the instructions from an internal register, an internal cache, the memory 608, or the storage device 610 and decode and execute them to execute instructions. In some embodiments, the at least one processor 606 includes one or more internal caches for data, instructions, or addresses. The at least one processor 606 may include one or more instruction caches, one or more data caches, and one or more translation look aside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in the memory 608 or the storage device 610.

[0107] The memory 608 may be coupled to the at least one processor 606. The memory 608 may be used for storing data, metadata, and programs for execution by the processor(s). The memory 608 may include one or more of volatile and non-volatile memories, such as Random- Access Memory ("RAM"), Read-Only Memory ("ROM"), a solid state disk ("SSD"), Flash, Phase Change Memory ("PCM"), or other types of data storage. The memory 608 may be internal or distributed memory.

[0108] The storage device 610 may include storage for storing data or instructions. As an example, and not by way of limitation, storage device 610 may include a non-transitory storage medium described above. The storage device 610 may include a hard disk drive (HDD), Flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. The storage device 610 may include removable or non-removable (or fixed) media, where appropriate. The storage device 610 may be internal or external to the storage device 610. In one or more embodiments, the storage device 610 is non-volatile, solid-state memory. In other embodiments, the storage device 610 includes read-only memory (ROM). Where appropriate, this ROM may be mask programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or Flash memory or a combination of two or more of these.

[0109] The storage device 610 may include machine-executable code stored thereon. The storage device 610 may include, for example, a non-transitory computer-readable storage medium. The machine-executable code includes information describing functional elements that may be implemented by (e.g., performed by) the at least one processor 606. The at least one processor 606 is adapted to implement (e.g., perform) the functional elements described by the machine-executable code. In some embodiments the at least one processor 606 may be configured to perform the functional elements described by the machine-executable code sequentially, concurrently (e.g., on one or more different hardware platforms), or in one or more parallel process streams.

[0110] When implemented by the at least one processor 606, the machine-executable code is configured to adapt the at least one processor 606 to perform operations of embodiments disclosed herein. For example, the machine-executable code may be configured to adapt the at least one processor 606 to perform at least a portion or a totality of the method 400 of FIG. 4 and / or the method 500 of FIG. 5. As another example, the machine-executable code may be configured to adapt the at least one processor 606 to perform at least a portion or a totality of the operations discussed for the sorter device 100 of FIG. 1. As a specific, non-limiting example, the machine-executable code may be configured to adapt the at least one processor 606 to causethe ML object sorter model training system 117 to train a ML object sorter model 252 and / or to cause the ML object sorter model 252 to sort a bulk product, as described above with reference to FIG. 1 through FIG. 5.

[0111] The input / output device 612 may allow an operator of the sorter device 100 to provide input to, receive output from, the computing device 602. The input / output device 612 may include a mouse, a keypad or a keyboard, a joystick, a touch screen, a camera, an optical scanner, network interface, modem, other known I / O devices, or a combination of such I / O interfaces. The input / output device 612 may include one or more devices for the operator to toggle between various displays of the output image.

[0112] In some embodiments, the bus 614 (e.g., a Controller Area Network (CAN) bus, an ISOBUS (ISO 11783 Compliant Implement Control)) may include hardware, software, or both that couples components of computing device 602 to each other and to external components.

[0113] All references cited herein are incorporated herein in their entireties. If there is a conflict between definitions herein and in an incorporated reference, the definition herein shall control.

[0114] While the present disclosure has been described herein with respect to certain illustrated embodiments, those of ordinary skill in the art will recognize and appreciate that it is not so limited. Rather, many additions, deletions, and modifications to the illustrated embodiments may be made without departing from the scope of the disclosure as hereinafter claimed, including legal equivalents thereof. In addition, features from one embodiment may be combined with features of another embodiment while still being encompassed within the scope as contemplated by the inventors. Further, embodiments of the disclosure have utility with different and various machine types and configurations.

Claims

CLAIMSWhat is claimed is:

1. A sorter device, comprising: an infeed system configured to provide a stream of a commodity proximate at least one detection system, the at least one detection system comprising at least one emitting device configured to emit electromagnetic radiation and at least one optical sensor configured to measure electromagnetic radiation reflected from the stream; an ejector configured to selectively expose the stream to air to sort the commodity; and a computing device operably coupled to the detection system and the ejector, the computing device comprising: at least one processor; and at least one non-transitory computer-readable storage medium storing instructions thereon that, when executed by at least one processor, cause the computing device to: cause an image processing system to generate image data of the stream based on the measured electromagnetic radiation; cause a communications interface to display an output image based on the image data; cause the communications interface to receive a user input indicating labeling data of at least some pixels defining objects of the commodity in the output image to generate training data; and train a machine learning object sorter model based on the training data.

2. The sorter device of claim 1, wherein the computing device comprises instructions thereon that, when executed by the at least one processor, cause the ejector to sort the commodity with the machine learning object sorter model.

3. The sorter device of claim 2, wherein the computing device comprises instructions thereon that, when executed by the at least one processor, cause the ejector to selectively expose objects of the commodity to air based on the machine learning object sorter model.

4. The sorter device of any one of claims 1 through 3, wherein the computing device comprises instructions thereon that, when executed by the at least one processor, cause the image processing system to perform a background thresholding operation on the image data to generate modified image data.

5. The sorter device of any one of claims 1 through 3, wherein the computing device comprises instructions thereon that, when executed by the at least one processor, cause the image processing system to perform one or more of a background thresholding operation, a contouring operation, an erosion operation, a growing operation, an area thresholding operation, and a feature extraction operation on the image data to generate modified image data.

6. The sorter device of claim 4 or claim 5, wherein the computing device comprises instructions thereon that, when executed by the at least one processor, cause the communications interface to display the output image based on the modified image data.

7. The sorter device of any one of claims 1 through 6, wherein the computing device comprises instructions thereon that, when executed by the at least one processor, cause the computing device to generate the training data based on a color of the commodity.

8. The sorter device of any one of claims 1 through 7, further comprising a machine learning object sorter system configured to: receive additional image data during a sorting operation; andcause an ejector controller to control the ejector based on the additional image data and the machine learning object sorter model.

9. The sorter device of claim 8, wherein the machine learning object sorter system comprises instructions thereon that, when executed by the at least one processor, cause the machine learning object sorter system to update the machine learning object sorter model based on additional user input.

10. The sorter device of any one of claims 1 through 9, wherein the computing device comprises instructions thereon that, when executed by at least one processor, cause a machine learning object sorter training system to train the machine learning object sorter model based on only a color of the at least some pixels defining objects of the commodity in the output image.

11. A sorter device, comprising: an infeed system configured to provide a stream of a commodity; at least one emitting device configured to emit electromagnetic radiation at the stream; at least one optical sensor configured to measure electromagnetic radiation reflected from the stream; an ejector configured to selectively expose the stream to air to sort the commodity; and a computing device operably coupled to the detection system and the ejector, the computing device comprising: at least one processor; and at least one non-transitory computer-readable storage medium storing instructions thereon that, when executed by at least one processor, cause the computing device to: cause an image processing system to generate image data of the stream based on the measured electromagnetic radiation;apply a machine learning object sorter model to the image data to generate a labeling decision for objects of the commodity in the stream; and control the ejector based on the labeling decision.

12. The sorter device of claim 11, wherein the computing device comprises instructions thereon that, when executed by the at least one processor, cause the computing device to generate the labeling decision based on a color of objects in the image data.

13. The sorter device of claim 11 or claim 12, wherein the computing device comprises instructions thereon that, when executed by the at least one processor, cause the computing device to modify the image data using one or more of a thresholding operation, an area thresholding operation, and a contouring operation.

14. The sorter device of any one of claims 11 through 13, further comprising a machine learning object sorter training system configured to generate the machine learning object sorter model based on user input received by a communications interface of the computing device.

15. A method of operating a sorter device, the method comprising: obtaining image data of a commodity within a sorter device; displaying, on a communications interface, an output image based on the image data; receiving, on the communications interface, a user input for each of a plurality of selected object in the output image, the user input indicating a label for at least some pixels of each of the plurality of selected objects; creating training data from the labels for the at least some pixels of each of the plurality of selected objects; training a machine learning object sorter model to generate a sorting decision for a given object based on an input image of the given object, wherein the machine learningobject sorter model is trained based on the labels and associated image data of the plurality of selected objects within the training data; and sorting the commodity with the trained machine learning sorter model.

16. The method of claim 15, further comprising performing one or more image processing operations on the image data prior to displaying the output image on the communications interface.

17. The method of claim 16, wherein performing one or more image processing operations on the image data comprises performing one or more of a background thresholding operation, a contouring operation, an erosion operation, a growing operation, an area thresholding operation, and a feature extraction operation on the image data to create modified image data.

18. The method of claim 17, wherein displaying the output image comprises displaying the output image based on the modified image data.

19. The method of any one of claims 15 through 18, wherein sorting the commodity with the trained machine learning sorter model based on the training set comprises selectively controlling operation of an ejector based on sorting decisions generated by the trained machine learning sorter model while continuously receiving additional image data.20 The method of any one of claims 15 through 19, wherein sorting the commodity with the machine learning sorter model comprises: receiving additional image data of a stream of the commodity within the sorter device; generating a labeling decision for objects of the commodity within the stream based on the additional image data using the machine learning object sorter model; and providing the labeling decision to an ejector controller.

Images