Method of classifying a seed using a classifier model

Abstract

A computer‑implemented method for classifying a seed. A primary image that shows multiple views per seed is segmented into segmented views. Views belonging to the same seed are identified by computing, for each view, a centroid coordinate along one image axis and grouping views whose centroid coordinates fall within a predefined range to form a seed instance. The seed instance is provided to a classifier model that outputs one or more defect categories, optionally with severity and / or probability scores. In video implementations, segmented views detected in successive frames are linked to the same seed by enforcing centroid‑coordinate continuity within a temporal window, and the linked views form the seed instance for classification.

Description

[0001] Method of classifying a seed using a classifier model

[0002] BACKGROUND

[0003] [1] Quality control of seeds is critical for the agricultural industry. High-quality seeds ensure better crop yields, higher resistance to diseases, and overall superior agricultural productivity.

[0004] [2] Conventionally, specialized analysts manually classify a sample of the seeds (e g. of grains or coffee beans) in order to analyse the seeds. The sample may e.g. comprise a few hundred grams of seeds (e.g. 350 grams), i.e. for instance 2000 coffee beans.

[0005] [3] However, such conventional manual methods of quality analysis for seeds are not only time-consuming but also inherently prone to human error. Each quality analyst must evaluate multiple samples every day, a process that requires both skill and intense focus, leading to fatigue and errors over time. This creates an inefficient workflow and potential inaccuracies in results, especially given the variability in the quality of seeds from different regions or harvests. Moreover, since the analysis process demands precision and consistency, the risk of subjective bias increases as analysts handle numerous samples in a day.

[0006] [4] The time-intensive nature of manual analysis can create bottlenecks, particularly in high-volume agricultural laboratories, where delayed results can hinder decision-making for farming practices, logistics, and market distribution. In an industry where timing is critical for planting, harvesting, and distribution, delays caused by slow or inaccurate quality assessments can have a ripple effect across the entire supply chain, reducing efficiency and potentially impacting quality of the products.

[0007] [5] Modern technologies such as machine learning, image processing, and automation offer promising alternatives to automate processes. Automated systems can analyze large quantities of seeds more quickly than manual methods, reducing human error and speeding up workflows. However, current approaches are not suitable to replace the manual work of specialized human analysts, in particular with regard to the task of reliably classifying seeds.

[0008] SUMMARY OF THE DISCLOSURE

[0009] [6] Currently, it remains desirable to overcome the aforementioned problems and in particular to provide a method and system for analysing, in particular, classifying seeds more reliably and in less time.

[0010] [7] Accordingly, the present disclosure relates according to a first aspect to a computer-implemented method of classifying a seed using a classifier model, comprising: feeding a seed instance into the classifier model, the seed instance showing views (e.g. segmented views) of different sides of the seed, wherein in response the classifier model outputs (e.g. predicts) a defect category of the seed (or at least one defect category, i.e. possibly also several defect categories, e g. up to defect categories).

[0011] [8] By providing such a method, it becomes possible to classify a seed (i.e. outputting a defect category of the seed) in a reliable and accurate manner. In particular, since the classification is based on seed instance showing views of different sides of the seed, the classifier model can make a decision which takes into account the whole surface of the seed, or at least as much surface as possible of the seed, e.g. 350 or 360 degrees in 2 dimensions. Moreover, using a classifier model for classifying the seed allows automating the classification task in a fast manner, which provides a high repeatability of the classification results.

[0012] [9] Desirably, the views of the seed comprised by the seed instance show together the whole (i.e. entire) surface of the seed. For example, the views may represent perspective images of the seed from different viewpoints.

[0013]

[0010] The classifier model may be an Al (artificial intelligence) model. The classifier model may be e.g. a trained deep-learning model. For example, the classifier model may comprise a CNN (convolutional neural network) which extracts features of the seed instance, in particular of a single segmented view of the seed instance. Moreover, the classifier model may comprise a classifier neural network (e.g. one or several layers on top of the CNN), which outputs defect categories based on the extracted features.

[0014]

[0011] The classifier model may be configured to perform a multi-label defect classification based on a fed seed instance (in short, the classifier model may be a multi-label classifier model). Accordingly, the output of the classifier model may comprise a multi-label structure. In other words, the model may accept several true answers (i.e. classes or labels) as output.

[0015]

[0012] Hence, the classifier model may assign multiple labels to a seed instance, allowing the seed instance to belong to more than one defect category simultaneously.

[0016]

[0013] The output layer of the classifier model may e g. comprise a dedicated sigmoid output for each possible label. Accordingly, compared to a conventional classifier model (e.g. a binary or multiclass classifier), the output layer (i.e. last activation function of the model) may be changed from a softmax output layer to a sigmoid output layer. Sigmoid may be suitable for multilabel classification because it allows for independent probabilities for each label, while softmax is designed for exclusive class assignments, such as in multiclass classification.

[0017]

[0014] For example, the classifier model may output up to two defect categories for a seed. For instance, the classifier model may output a major defect category, e.g. based on a probability of the defect category determined by classifier model, the probability optionally being above a predefined threshold and being the highest compared to the determined probabilities of all other defect categories. Furthermore, the classifier model may output a minor defect category, e.g. based on a probability of the defect category determined by classifier model, the probability optionally being above a predefined threshold and being the second highest compared to the determined probabilities of all other defect categories.

[0018]

[0015] Due to the optional predefined threshold it may also be possible, that the classifier model only outputs a major defect category or no defect category for a seed. However, in case the defect categories on which the classifier model is trained comprise also categories of healthy (i.e. undefected seeds), the threshold may be not necessary. In this case, if a category of a healthy seed has the highest probability for each view of the seed, the classifier model may output no defect category for a seed, i.e. that the seed is “good” or “healthy”.

[0019]

[0016] The classifier model may also comprise several machine-learning and / or deep-learning models. For example, the classifier model may comprise a first model which is trained to detect anomalies based on the seed instance. Accordingly, the first model may be trained to distinguish seeds comprising no defect (i.e. “healthy” seeds) from seeds comprising potentially one or several defects (i.e. defected or suspicious seeds). The first model may hence be e.g. a bi-classifier model.

[0020]

[0017] Furthermore, the classifier model may comprise a second model which is trained to output a defect category of a seed, in particular of a view of the seed. In particular, the second model may be fed only with seed instances of defected, i.e. suspicious seeds, but not with seed instances of healthy seeds. The second model may hence comprise the CNN and classifier neural network, as described above. The second model may further be the multi-label classifier model as described above.

[0021]

[0018] The first model may be e.g. a relatively small and fast model compared to the second model. Accordingly, the speed of the method and / or its computational costs may be reduced by using the first and second models, since only the seed instances of defected seeds are analysed by the second model.

[0022]

[0019] The seed may be e.g. a coffee bean, e.g. an unroasted coffee bean. In addition to coffee beans (particularly before being roasted), there are numerous other seeds and grains that can benefit from segmentation and classification methods to identify potential defects.

[0023] Here are a few examples:

[0024]

[0020] Wheat Grains: Wheat is a staple crop worldwide, and its quality directly impacts the production of flour and bread. Defects in wheat grains such as fungal contamination, sprouting, or physical damage can affect its usability and yield. Segmentation and classification methods can isolate individual grains and detect specific defects, such as decolored kernels or damage caused by pests.

[0025]

[0021] Rice Grains: Rice, being a global dietary staple, is often classified based on shape, size, and color uniformity. Defects such as cracked or broken rice grains, contamination with foreign particles, or decoloration can degrade the quality of rice. Segmentation can separate each grain for detailed analysis, ensuring that only quality grains are selected, and defective grains are categorized accordingly.

[0026]

[0022] Soybeans: Soybeans are used for a variety of products, including tofu, soy milk, and animal feed. Defects such as splits, shriveling, or mold growth can reduce the quality of the crop. By using segmentation to isolate individual beans, each can be analyzed for such defects, which ensures higher quality in downstream processes.

[0027]

[0023] Corn Kernels: Corn is used for food, biofuel, and industrial products, and its quality must be monitored to ensure optimal use. Defects in corn kernels, such as damage due to mold, insect infestation, or poor kernel development, can be effectively identified using segmented views. This helps in determining whether the kernels are fit for processing or sale.

[0028]

[0024] Barley Grains: Used primarily for brewing and animal feed, barley is graded based on factors like color, size, and the presence of foreign matter. Segmentation can help identify kernels that are decolored, shriveled, or affected by fungal infections, leading to a more accurate assessment of barley quality.

[0029]

[0025] Sunflower Seeds: Often used for oil production and as a snack, sunflower seeds require strict quality control. Segmentation can help detect defects such as shell damage, underdeveloped seeds, or the presence of foreign matter, ensuring that only high-quality seeds are used for oil extraction or sold as edible snacks.

[0030]

[0026] Peanuts: As peanuts are used in various products, including peanut butter and cooking oil, detecting defects like aflatoxin contamination, shell damage, or shriveled nuts is critical. Segmentation allows individual peanuts to be inspected in detail, ensuring that only safe and high-quality peanuts are processed.

[0031]

[0027] Chia Seeds: Chia seeds, which are popular for their nutritional benefits, must be free of contaminants and defects. Segmentation can help distinguish between high-quality seeds and those that are damaged, underdeveloped, or contaminated with foreign particles.

[0032]

[0028] Cocoa beans and tree nuts (almonds, hazelnuts, pistachios, cashews, pecans, walnuts, macadamia nuts, Brazil nuts and pine nuts): As with other seeds and grains, the disclosed segmentation and multi-view classification pipeline isolates each nut or bean and enables detection of defects (e.g., shell damage, kernel discoloration, insect damage, mold), improving reliability and throughput.

[0029] The computer-implemented method may be carried out by a computing system. For example, the classifier model and optionally any other software modules or computer algorithms be running on (i.e. performed or executed by) the computing system.

[0033]

[0030] The seed instance may comprise at least one of:

[0034] a primary image showing the views, wherein the views are annotated with respective segmentation masks,

[0035] a single secondary image having a single canvas on which the segmented views are placed adjacent to each other, and

[0036] a set of secondary images, wherein for each view a respective secondary image has a canvas, on which the segmented view is placed.

[0037]

[0031] A primary image may e.g. result from an imaging system (e.g. a camera system), and may have been annotated with respective segmentation masks by a segmentation module. The primary image may also result from e g. a video, e.g. from one or several image frames of the video.

[0038]

[0032] Accordingly, the primary segmented image (as taken e.g. by a camera system) may be fed into the classifier model. The segmentation mask may e.g. be used to “neutralize”, filter out or cut off all background and views of potential other seeds on the primary image.

[0039]

[0033] Segmenting the views of a seed before feeding them into a classifier model offers several key advantages in terms of accuracy, precision, and efficiency when classifying defect categories.

[0040]

[0034] First, segmentation ensures that only relevant parts of the image corresponding to an individual seed are analyzed, effectively isolating the seed from its surroundings or from other overlapping seeds. This reduces the chance of misclassification due to noise or interference from adjacent seeds, especially in cases where seeds may be clustered or positioned too closely together

[0041]

[0035] Second, by working with segmented views, the classifier model receives a more detailed and localized perspective of each seed, allowing it to focus on specific features and defects that may be unique to that seed. This helps the model identify subtle imperfections or quality indicators that might be overlooked in a broader, unsegmented image.

[0042]

[0036] Third, multiple segmented views from different angles or parts of the same seed can provide richer information to the classifier. By combining these views, the model can generate a more comprehensive understanding of the seed’s characteristics, leading to more robust and reliable defect detection.

[0043]

[0037] The secondary image(s) may be obtained by processing the primary image, e.g. by placing the segmented views on a single canvas or each view on a respective canvas.

[0038] The defect category may indicate a particular defect of the seed. The outputted defect category may further comprise at least one of a severity score indicating the severity of a defect category (e.g. an extent / severity of a defect of a particular type), and a probability score indicating the likelihood of a defect category.

[0044]

[0039] Examples of defect types (i.e. of possible defect categories) of coffee beans (one example of seeds) comprise e g.:

[0045]

[0040] Primary defect types, such as:

[0046] Full Black,

[0047] Full Sour,

[0048] Cherry / Pod,

[0049] Mold,

[0050] Foreign matter,

[0051] Severe Insect,

[0052] Secondary defect types, such as:

[0053] Part Black,

[0054] Part Sour,

[0055] Pergamino,

[0056] Parchment,

[0057] Floater,

[0058] Immature,

[0059] Withered,

[0060] Shell,

[0061] Broken,

[0062] Hull / Husk,

[0063] Part Insect, and

[0064]

[0041] Additional defect types, such as:

[0065] Flat / Crushed,

[0066] Crystallized,

[0067] Decolored veined,

[0068] Decolored reposado,

[0069] Decolored amber or montequilla. Decolored overdried,

[0070] Wet (“wet” may be considered as an additional defect type or as a category of a healthy seed),

[0071] Stick (i.e. a foreign matter shown by the seed instance, e.g. a stick),

[0072] Stone (i.e. a foreign matter shown by the seed instance, e.g. a stone).

[0042] Moreover, the defect categories may comprise at least one category for a healthy seed, i.e. no defect. Examples include:

[0073] “good” (i.e. no other defect type recognized), foxy, peaberry, decolored, and wet.

[0074]

[0043] Categories like "foxy", “peaberry” “decolored”, and “wet” are characterised as “good”. These categories help to reduce false positives. In other words, since the classifer model may be trained on a training dataset which also comprise samples of actual healthy seeds (which do not comprise an actual defect, but anyway comprise visual characteristics like “foxy”), the classifier model become more robust against outputting false positive (i.e. recognizing defect categories for an actually healthy seed).

[0075]

[0044] The “defect category” may also be referred to as a “seed category”, as it may also comprise non-defected categories, as explained above.

[0076]

[0045] The method may further comprise, before feeding the seed instance into the classifier model: feeding a primary image into a segmentation module, the primary image showing, for at least one seed, views of different sides of the seed.

[0077]

[0046] The segmentation module may be used for at least one of (in particular, in response to the fed primary image, the segmentation module may perform at least one of): segmenting the primary image into a plurality of segmented views of the seed, and annotating each view of the plurality of segmented views with a segmentation mask on the primary image.

[0078]

[0047] A seed instance may be determined by at least one of:

[0079] annotating the primary image with the segmentation masks, and

[0080] placing the segmented views on a common canvas or placing each segmented view on an individual canvas.

[0081]

[0048] The determination of the seed instance may be done by e.g. the segmentation module or by another software module or algorithm running on the computing system.

[0082]

[0049] Generally, the segmentation module, the pre-processing module, the classifier model and any other software module or algorithm mentioned in this disclosure may be running on the computing system i.e. may be performed by the computing system.

[0083]

[0050] The method may further comprise: identifying, among the plurality of segmented views, a set of segmented views relating to the same seed based on the relative positions of the segmented views in the primary image.

[0084]

[0051] For example, based on the identified set of segmented views, it may be determined, whether the primary image shows views of different seeds, e.g. three views of a first seed and three views of a second seed.

[0085]

[0052] The relative positions of the segmented views may comprise the offsets of the centroids of the segmented views to each other in one dimension of the primary image.

[0053] The primary image may be a 2D image having two dimensions, e.g. along perpendicular X and Y axis.

[0086]

[0054] The secondary image or images may also be each a 2D image having two dimensions, e.g. along perpendicular X and Y axis.

[0087]

[0055] Determining the set may comprise: for each segmented view of the plurality of segmented views, determining a level of a centroid of the segmented view in one dimension the primary image,

[0088] comparing the levels of the centroids, and

[0089] adding the segmented views to the set whose levels of the centroids are similar or within a predefined level range.

[0090]

[0056] The determined level of the centroid may also be or be represented by an offset of the centroid from a reference point (e.g. the origin of a coordinate system of the primary image).

[0091]

[0057] Accordingly, the views may be oriented and / or positioned such on the primary image that the views relating to the same seed are always in the same level. This may be e g. due to geometric characteristics of the used camera system.

[0092]

[0058] The centroid and in particular its level on the first image may be determined using a predefined computer algorithm (e.g. computer vision algorithm). For example, the centroid and in particular its level may be determined by the computer algorithm based on a segmented view or its segmentations mask.

[0093]

[0059] Said computer algorithm may further identify the set of segmented views relating to the same seed. On this basis, the computer algorithm may determine, whether the primary image shows views of different seeds, e g three views of a first seed and three views of a second seed.

[0094]

[0060] Said computer algorithm may thus also be referred to in the following as

[0095]

[0061] Determining a seed instance may comprise at least one of:

[0096] assigning the segmented views of the set to the seed instance,

[0097] annotating the primary image with the segmentation masks of the views of the set, and placing the segmented views of the set on a common canvas or placing each segmented view of the set on an individual canvas.

[0098]

[0062] The viewing sides of the seed may comprise a front side, a back left side and a back right side.

[0099]

[0063] The primary image may be obtained by an imaging system comprising a mirror system which reflects the back left side and the back right side, such that an imaging sensor of the imaging system captures views of the main side, the reflected back left side and the reflected back right side.

[0064] Accordingly, each view may be specified by a particular / distinct viewing side of a particular / distinct seed, e.g. of one seed among a plurality of seeds shown on the primary image.

[0100]

[0065] The type of viewing side may be detected based on the position of the respective view on the primary image. For example, the view in the center of the primary image may be detected as (or assigned as) front side, the view on the left of the primary image as a back left side, and the view on the right of the primary image as a back right side.

[0101]

[0066] Feeding the seed instances into the classifier model may comprise: feeding the secondary images of the set of secondary images successively or in parallel into the classifier model.

[0102]

[0067] The input layer of the classifier model may be designed to receive the secondary images of the set of secondary images in parallel.

[0103]

[0068] The classifier model may be designed to process the secondary images of the set of secondary images successively or in parallel to output a class.

[0104]

[0069] The segmentation module may comprise at least one of:

[0105] a pre-processing module for at least one of: correcting lens distortion of the primary image, cropping peripheral areas of the primary image, and isolating the views from the background and / or foreground,

[0106] a computer vision algorithm, optionally comprising a trained machine-learning based segmentation model, for at least one of: edge detection, contour finding, watershed segmentation to separate each view in the primary image from its background and / or foreground, and recognizing predefined patterns and features in the primary image, a trained deep-learning based segmentation model for segmenting the primary image, optionally pixel by pixel, for distinguishing each seed instance from its background or foreground, and optionally identifying for each seed instance a predefined defect category.

[0107]

[0070] Each view in the primary image may be separated from its background and foreground, in order to separate the view of a first seed e.g. from an overlapping view of second seed. For example, in case two seeds are placed (accidentally) too close to each other in the camera system, one seed (i.e. at least one of its views) may partially hide another seed (i.e. at least one of its views) in the primary image.

[0108]

[0071] For instance, seeds can be positioned too close to each other (less than 1 cm apart) in the camera system, leading to what is referred to as a "seed trail." This occurs when multiple seed are aligned or clustered together, which can cause issues during processing or analysis, i.e. classification of a seed. In some cases, the seeds may even be touching one another, forming a "seed cluster." These clusters or trails make it difficult to isolate and evaluate individual seeds, potentially affecting the accuracy of the quality control processes (i.e. defect classification process), such as defect detection or grading, as well as disrupting the uniformity in downstream operations like packaging or roasting.

[0109]

[0072] In order to address this problem the views of the seeds (e.g., coffee beans) may be segmented in the primary image. By applying image segmentation, each seed may be accurately isolated, separating the visual information related to different seeds. Through segmentation, views of individual seeds may be grouped into distinct sets of segmented views, ensuring that each set corresponds to a single seed.

[0110]

[0073] This method ensures that only the segmented views of the same seed may be fed into the classifier model. As a result, the classification process is not affected by overlapping or touching seeds, preventing any misinterpretation of features that could arise from bean clusters or "seed trails." By handling each seed individually, the classifier can focus on accurately assessing the quality, detecting defects, or grading the characteristics of each seed. This approach enhances the accuracy and reliability of the classification process, eliminating noise or confusion caused by closely positioned or clustered beans.

[0111]

[0074] For example, the relatively fast and computationally less complex computer vision algorithm may be used for segmenting the views in the primary image in the method of the present disclosure. The computer vision algorithm may e.g. use or comprise a conventional machine-learning model, but no deep-learning model.

[0112]

[0075] Once the views are segmented, the primary image being annotated with respective annotations masks may be fed to the multi-seed detection algorithm.

[0113]

[0076] The trained deep-learning based segmentation model may be computationally more complex and / or computationally more expensive than the computer vision algorithm. The trained deep-leaming based segmentation model may e.g. be used (only) for evaluating the accuracy and / or reliability of the computer vision algorithm. However, it may also be used for the segmentation task, i.e. replace the computer vision algorithm.

[0114]

[0077] The classifier model may be trained to classify the seed, based on the seed instance, into at least one predefined defect category.

[0115]

[0078] The classifier model may determine based on each view of the seed instance at least one defect category or up to two (or a predefined number which is more than two) defect categories. The classified defect categories of the seed may comprise a sum of the defect categories determined for the views of the respective seed.

[0116]

[0079] Generally, the classified defect categories of the seed may be determined based on the defect categories determined for the views of the respective seed.

[0080] For example, in case a defect category determined for any one of the views of a seed instance is above a predefined threshold, this defect category may be assigned to the seed (i.e. on seed instance level). Optionally, only up to two defect categories for each side may be assigned to the seed, e.g. those having the highest probability. In case more than two defect categories are assigned to the seed, the two defect categories with the highest probabilities may be maintained, e.g. as major defect category and minor defect category.

[0117]

[0081] In case a defect category determined for a first view of a seed instance is a “healthy” category but for any other view of said seed instance an actual defect category is determined, the healthy category is discarded on seed instance level. Accordingly, if a seed shows on one side an actual defect and on another not defect, the seed may anyway be classified as having a defect. Only in case for all views of a seed instance, a “healthy” category is determined with having the highest probability for each view, the seed may be classified as undefected.

[0118]

[0082] The method further may comprise feeding a set of primary images related to a plurality of seeds to the segmentation model which outputs in response respective seed instances (e g. a set of seed instances) related to plurality of seeds.

[0119]

[0083] The classifier model may process the seed instances in parallel by using multiple parallel processing streams executed by one or more processing units, such as GPUs, to output predicted classes of the different seeds.

[0120]

[0084] The method further may comprise:

[0121] feeding a set of seed instances related to different seeds into the classifier model, wherein in response the classifier model outputs a defect category of the seed for each seed instance, optionally weighting the defect category according to a predefined weighting scheme or metric, and

[0122] assigning one or several defect categories to the set of seed instances based on the outputted and optionally weighted defect categories.

[0123]

[0085] The set of seed instances may be based on a set of primary images, as described above. For example, the set of seed instances may relate to a selection of seeds (e.g. 1000 to 3000 seeds) taken out of a larger seed collection (e.g. a harvest volume or a sold volume of seeds, e.g. 60kg of seeds).

[0124]

[0086] The set of seed instances may be used to evaluate the quality of the larger seed collection, e g. to detect potential defect categories in the larger seed collection. In other words, the larger seed collection may be represented by the set of seed instances.

[0125]

[0087] For this purpose the method according to the present disclosure may be applied to each seed instance of the set of seed instances, in order to output a defect category for each seed instance.

[0088] Since some defect categories are more severe than others (cf. the primary, secondary and additional defect types, as described above), the defect categories may be weighted according to a predefined weighting scheme or metric. For example, the primary, secondary and additional defect types may be weighted with different predefined weights, e.g. a primary defect type with a weight of 1, a secondary defect type with a weight of 0.3 or 0.2 and an additional defect type with a weight of 0.1. Accordingly, less severe defect types, such as secondary or additional defect types need to occur more often (e.g. 10 times more) to be considered as having the same relevance in the set of seed instances (cf. the example of assigning top n defect categories to the set of seed instances, as described below).

[0126]

[0089] The set of seed instances (and thus optionally also the larger seed collection) may be assigned with one or several defect categories based on the outputted and optionally weighted defect categories. For example, the sum of all defect categories may be assigned to the set of seed instances (and thus optionally also the larger seed collection), optionally after weighting the defect categories. Alternatively, only the top n defect categories (of all outputted defect categories) may be assigned to the set of seed instances (and thus optionally also the larger seed collection), optionally after weighting the defect categories, n being a natural number. The top n defect categories may be determined based on the number of occurrences of particular defect categories, which have been outputted for the seed instances, optionally after weighting the defect categories.

[0127]

[0090] The present disclosure further relates to a method of generating training samples for an image segmentation and classification module, the image segmentation and classification module comprising an image segmentation module and an image classifier model, the method comprising:

[0128] providing a set of primary images, each primary image showing, for at least one seed, views of different sides of the seed,

[0129] labelling each view by at least one of:

[0130] assigning a segmentation mask to the view,

[0131] assigning a seed identifier to the view which distinguishes the seeds shown in the primary image,

[0132] assigning a type of viewing side from a predefined number of viewing sides to the seed instance, and

[0133] assigning at least one defect category to the view, the defect category indicating a defect of the seed, the defect category optionally further comprising at least one of a severity score indicating the severity of a defect type, and a probability score indicating the likelihood of a defect type.

[0091] The image classifier model may be or may comprise the classifier model as described above.

[0134]

[0092] The image segmentation module may e.g. comprise or may be the above-mentioned computer vision algorithm (comprising optionally a machine-learning based segmentation model) and / or the deep-learning based segmentation model.

[0135]

[0093] The image segmentation and classification module may comprise further modules or software algorithms, e.g. the pre-processing module, as described above.

[0136]

[0094] The defect categories may be assigned based on the characteristics of the defect categories, as described above. For example, the defect categories may comprise any category as described above, comprising e.g. also “healthy” categories. Moreover, each viewing side may be assigned with up to two defect categories, e.g. labelled as major defect category and minor defect category, as described above. In short, the training dataset may be such that the trained classifier model has the characteristics, as described above.

[0137]

[0095] The present disclosure further relates to a method of training an image segmentation and classification module, the image segmentation and classification module comprising an image segmentation module and an image classifier model, the method comprising: providing training samples generated by the method as described above,

[0138] training the image segmentation and classification module in a supervised manner using the training samples.

[0139]

[0096] Training the image segmentation and classification module may in particular comprise or be limited to training the classifier model. Hence, the image segmentation module may not be trained, but could be trained, as well, based on the training dataset. Therefore, the method of training an image segmentation and classification module may also be a method of training a classifier model

[0140]

[0097] Training the classifier model may comprise fine tuning the classifier model,

[0141]

[0098] The method may further comprise:

[0142] running the trained image segmentation and classification module by feeding a set of customized primary images into the module,

[0143] identifying incorrect defect categories among the outputted defect categories,

[0144] generating a customized training dataset by assigning secondary samples having an incorrect defect category with a correct defect category, the secondary samples being outputted by the segmentation module in response to feeding respective primary images into the segmentation module,

[0145] retraining the classifier model using the corrected secondary samples.

[0099] By running the trained image segmentation and classification module on a set of customized primary images, the process enables the identification of incorrect defect categories that might arise during initial classifications. This allows for a deeper level of quality control, as errors can be systematically detected and corrected, and in particular to adapt the module to customized seed characteristics, e.g. of a particular processing type of the seeds or origin.

[0146]

[0100] The key advantage lies in the generation of a customized training dataset, where secondary samples, which have been misclassified, are reassigned to their correct defect categories. This ensures that the system learns from its mistakes, enhancing the accuracy of future classifications. The process of retraining the classifier model using these corrected samples continuously refines the model's ability to distinguish between different defect categories with greater precision.

[0147]

[0101] In essence, this iterative correction and retraining process creates a dynamic feedback loop, allowing the classifier to evolve and adapt to complex or rare defect patterns that may not have been accounted for during the initial training. Over time, this leads to a highly tailored and fine-tuned system that minimizes classification errors, ensures more reliable outcomes, and boosts the overall efficiency of seed quality control processes. The adaptability of the method is particularly advantageous in agricultural environments, where variability in seed quality can be substantial across different batches and environments.

[0148]

[0102] In general, the method of the present disclosure may be performed by a computing system. Thus, the method may be a computer-implemented method. In case the method steps comprise any actions (for example physical actions) which go beyond a mere data processing operation (for example controlling the camera system), the computer-implemented method may further comprise instructions which, when executed by the computing device, cause the computing system to carry out these actions.

[0149]

[0103] The computing system may comprise or may be associated with a user interface (e.g. provided by a computer device connected to the computing system).

[0150]

[0104] The user interface may be provided e.g. by a personal computer, a smartphone, tablet or any other computer device having a user interface, e g. via a respective app running on the smartphone. The user interface may be a graphical user interface running display on a e g. a display screen.

[0151]

[0105] For example, the user input may be provided to the computing system via the user interface, and / or the output may be provided to a user via the user interface.

[0152]

[0106] The computing system may at least partially use cloud resources, e.g. cloud servers.

[0107] A second aspect of the present disclosure relates to a computing system (i.e. a data processing system) comprising a processor configured to carry out the method according to the first aspect of the present disclosure. For example, the computing system may further comprise at least one memory storing computer-executable instructions, the computer-executable instructions when executed by the processor cause the computing device and / or its processor to carry out the method according to the present disclosure.

[0153]

[0108] The processor (or processing unit) may be a component of electronic devices that may be responsible for carrying out computational tasks. There may be different types of processing units, each designed for specific purposes. A processing unit may be or may comprise a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), and / or Application-Specific Integrated Circuit (ASIC).

[0154]

[0109] A third aspect of the present disclosure relates to a computer program product comprising instructions which, when the program may be executed by a computing system, cause the computing system to carry out the method according to the present disclosure.

[0155]

[0110] A fourth aspect of the present disclosure relates to a computer-readable medium, e.g., a non-transitory computer-readable medium, comprising instructions which, when executed by a computing system, cause the computing system to carry out the method according to the first aspect of the present disclosure.

[0156]

[0111] It is intended that combinations of the above-described elements and those within the specification may be made, except where otherwise contradictory.

[0157]

[0112] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only, are provided for illustration purposes and are not restrictive for the scope as defined by the accompanied claims.

[0158]

[0113] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples of the disclosure and together with the description, and serve to support and illustrate the principles thereof.

[0159] BRIEF DESCRIPTION OF THE DRAWINGS

[0160]

[0114] Fig. 1 shows a schematic exemplary representation of the method of the present disclosure.

[0161]

[0115] Fig. 2 shows a schematic exemplary representation of an imaging system (e.g. a camera system) to obtain primary images according to the present disclosure.

[0162]

[0116] Fig. 3a shows a first example of a primary image according to the present disclosure.

[0117] Fig. 3b shows a second example of a primary image according to the present disclosure.

[0163]

[0118] Fig. 4 shows a schematic exemplary representation of a computing system on which software modules according to the present disclosure are running.

[0164] DESCRIPTION OF THE DRAWINGS

[0165]

[0119] In the following text, a detailed description of examples will be given with reference to the drawings. Various modifications to the examples may be made. In particular, one or more elements of one example may be combined and used in other examples to form new examples.

[0166]

[0120] Fig. 1 shows a schematic exemplary representation of the method 100 of the present disclosure

[0167]

[0121] In an optional step SI a primary image may be fed into a segmentation module.

[0168]

[0122] The primary image may e.g. result from an imaging system (e.g. camera system), and may have been annotated with respective segmentation masks by a segmentation module. The primary image may also result from e g. a video, e.g. from one or several image frames of the video.

[0169]

[0123] Fig. 2 shows a schematic exemplary representation of camera system 200 to obtain primary images according to the present disclosure.

[0170]

[0124] Fig. 3a shows a first example of a primary image 300a according to the present disclosure.

[0171]

[0125] Fig. 3b shows a second example of a primary image 300b according to the present disclosure.

[0172]

[0126] The primary image may show different views 301, 302, 303 (cf. fig. 3a) of a seed 202 (cf. fig. 2), e.g. a coffee bean. The viewing sides of the seed may comprise a front side, a back left side and a back right side.

[0173]

[0127] The primary image may be obtained by an imaging system 200 comprising an imaging sensor 201 (e.g. a digital camera or other optical sensor or imaging sensor) and a mirror system 204a, 204b which reflects the back left side and the back right side. In this way, the imaging sensor can capture views of the main side, the reflected back left side and the reflected back right side of the seed 202.

[0174]

[0128] The seed 202 may be placed by a seed feeding device (not shown) on a transparent rotating plate 203 (e.g. a glass, disc), such that primary images of a plurality of seeds may be automatically captured by rotating the plate. A seed removal device (not shown) may remove the seed once its primary image has been taken.

[0175]

[0129] Desirably, the views of the seed comprised by the seed instance show together the whole (i.e. entire) surface of the seed. For example, the views may represent perspective images of the seed from different viewpoints.

[0176]

[0130] Optionally, the camera system may comprise several mechanical components (partially not shown):

[0177] • Seed Distribution Mechanism: Dispenses with a controlled speed a single or several seeds and grains one after one separated from each other onto a rotating disk.

[0178] • Rotating Disk 203: Transports a single or several seeds and grains one after one separated from each other from the seed distribution system to the photographing area in a consistent and controlled manner ensuring the seeds and grains are not changing their position on a rotating disk and follow a predefined trajectory.

[0179] • Light Cone: Provides a uniform illumination on the photographing area to light up the front side of beans and to avoid shadows under seeds and grains and reflections during image capturing. Light Cone is installed above an imaging sensor and contains multiple LEDs to achieve the optimal LUX and K for the imaging sensor to be able to capture the at the highest shutter speed possible for this lighting set up.

[0180] • Mirror System 204a, 204b and LED Stripes: Reflects the back left and right side of the seeds and grains into the imaging sensor. Mirror system is installed below the Rotating Disk to reflect the seeds and grains as they pass the photographing area. LED stripes provide optimal LUX and K for the imaging sensor to be able to capture the reflections of seeds and grains at the optimal shutter speed.

[0181] • Uniform Background: Ensures a consistent background reflected into the imaging sensor for front, back left and back right images. Blue background stripes are attached on the inner walls of the Light Cone on the right and left sides creating clear, high-contrast images that facilitate accurate segmentation of seeds and grains during pre-processing.

[0182] • Imaging Sensor 202 and Shutter Mechanism: Shutter mechanism detects the edge of the beans passing through the photographing area on the rotating disk and triggers an imaging sensor. Imaging Sensor captures images when seeds and grains are optimally placed in the middle of the photographing area in front of the imaging sensor.

[0183] • Vacuum Cleaning System: Wipes the rotating disk and removes fat, dust, debris to maintain image quality. Vacuum cleaning system including a roll and a vacuum cleaner is installed above the rotating disk.

[0131] An optional microcontroller may ensure that key mechanical components operate in a synchronized manner, activating, operating, and deactivating each part of the image capturing system according to achieve the high-quality images efficiently. The results of this image capture process is a standardized high-resolution image of a single or several seeds or grains in the middle of the frame captured from front, back left and back right sides on a uniform background.

[0184]

[0132] In an optional step S2 the primary image may be segmented into a plurality of segmented views of the seed using the segmentation module

[0185]

[0133] The segmentation module may be used for at least one of (in particular, in response to the fed primary image, the segmentation module may perform at least one of): segmenting the primary image into a plurality of segmented views of the seed, and annotating each view of the plurality of segmented views with a segmentation mask on the primary image.

[0186]

[0134] The segmentation module may comprise at least one of:

[0187] a pre-processing module for at least one of: correcting lens distortion of the primary image, cropping peripheral areas of the primary image, and isolating the views from the background and / or foreground,

[0188] a computer vision algorithm, optionally comprising a trained machine-learning based segmentation model, for at least one of: edge detection, contour finding, watershed segmentation to separate each view in the primary image from its background and / or foreground, and recognizing predefined patterns and features in the primary image, a trained deep-learning based segmentation model for segmenting the primary image, optionally pixel by pixel, for distinguishing each seed instance from its background or foreground, and optionally identifying for each seed instance a predefined defect category.

[0189]

[0135] Each view in the primary image may be separated from its background and foreground, in order to separate the view of a first seed e.g. from an overlapping view of second seed. For example, in case two seeds are placed (accidentally) too close to each other in the camera system, one seed (i.e. at least one of its views) may partially hide another seed (i.e. at least one of its views) in the primary image.

[0190]

[0136] For instance, as shown in fig. 3b, seeds can be positioned too close to each other (less than 1 cm apart) in the camera system, leading to what is referred to as a "seed trail." This occurs when multiple seeds are aligned or clustered together on the rotating plate, which can cause issues during processing or analysis, i.e. classification of a seed. In some cases, the seeds may even be touching one another, forming a "seed cluster." These clusters or trails make it difficult to isolate and evaluate individual seeds, potentially affecting the accuracy of the quality control processes (i.e. defect classification process), such as defect detection or grading, as well as disrupting the uniformity in downstream operations like packaging or roasting.

[0191]

[0137] As shown in the example of fig. 3b, a primary image 300b may thus show views of different seeds, e.g. views 301, 302, 303 of a first seed, and views 304, 305, 306 of a second seed.

[0192]

[0138] In order to address this problem the views of the seeds (e.g., coffee beans) may be segmented in the primary image. By applying image segmentation, each seed may be accurately isolated, separating the visual information related to different seeds. Through segmentation, views of individual seeds may be grouped into distinct sets of segmented views, ensuring that each set corresponds to a single seed.

[0193]

[0139] In an optional step S3 a set of segmented views relating to the same seed, among the plurality of segmented views relating potentially to more than one seed, may be identified.

[0194]

[0140] For example, based on the identified set of segmented views, it may be determined, whether the primary image shows views of different seeds, e.g. three views of a first seed and three views of a second seed.

[0195]

[0141] The relative positions of the segmented views may comprise the offsets 307a, 307b of the centroids of the segmented views to each other in one dimension of the primary image.

[0196]

[0142] The primary image may be a 2D image having two dimensions, e.g. along perpendicular X and Y axis.

[0197]

[0143] The secondary image or images may also be each a 2D image having two dimensions, e.g. along perpendicular X and Y axis.

[0198]

[0144] Determining the set may comprise:

[0199] for each segmented view 301 to 306 of the plurality of segmented views, determining a level 307a, 307b of a centroid of the segmented view in one dimension the primary image, comparing the levels 307a, 307b of the centroids, and

[0200] adding the segmented views (e.g. 301, 302, 303) to the set whose levels of the centroids are similar or within a predefined level range (e.g. a predefined range around level 307a).

[0201]

[0145] The determined level of the centroid may also be or be represented by an offset of the centroid from a reference point (e.g. the origin of a coordinate system of the primary image).

[0202]

[0146] Accordingly, the views may be oriented and / or positioned such on the primary image that the views relating to the same seed are always in the same level. This may be e.g. due to geometric characteristics of the used camera system.

[0203]

[0147] This method ensures that only the segmented views of the same seed may be fed into the classifier model. As a result, the classification process is not affected by views of different seeds, preventing any misinterpretation of features that could arise from seed clusters or "seed trails." By handling each seed individually, the classifier can focus on accurately assessing the quality, detecting defects, or grading the characteristics of each seed. This approach enhances the accuracy and reliability of the classification process, eliminating noise or confusion caused by closely positioned or clustered beans.

[0204]

[0148] In an optional step S4 a seed instance may be determined by:

[0205]

[0149] S4a: assigning the segmented views of the set of views to the seed instance,

[0206]

[0150] S4b: annotating the primary image with the segmentation masks of the views of the set,

[0207]

[0151] S4c: placing the segmented views of the set on a common canvas or placing each segmented view of the set on an individual canvas.

[0208]

[0152] Accordingly, the seed instance may comprise at least one of:

[0209] a primary image showing the views, wherein the views are annotated with respective segmentation masks,

[0210] a single secondary image having a single canvas on which the segmented views are placed adjacent to each other, and

[0211] a set of secondary images, wherein for each view a respective secondary image has a canvas, on which the segmented view is placed.

[0212]

[0153] The secondary image(s) may be obtained by processing the primary image, e.g. by placing the segmented views on a single canvas or each view on a respective canvas.

[0213]

[0154] In a step S5 the seed instance is fed into the classifier model. Feeding the seed instance into the classifier model may comprise feeding e.g. the secondary images of the set of secondary images successively or in parallel into the classifier model.

[0214]

[0155] For example, the input layer of the classifier model may be designed to receive the secondary images of the set of secondary images in parallel.

[0215]

[0156] The classifier model may be designed to process the secondary images of the set of secondary images successively or in parallel to output a class. Alternatively, the segmented primary image may be fed into the classifier model.

[0216]

[0157] In step S6 a defect category of the seed may be outputted by the classifier model. The defect category may indicate a particular defect of the seed. The outputted defect category may further comprise at least one of a severity score indicating the severity of a defect category (e g. an extent / severity of a defect of a particular type), and a probability score indicating the likelihood of a defect category.

[0217]

[0158] The classification of defect categories of a seeds, such as a coffee bean, may comprise primary, secondary, and additional defect types, with each category representing different levels of severity or types of imperfections, exemplary defect types include:

[0159] Primary Defect Types:

[0218] Primary defects are the most severe and significantly affect the quality of the seed. They are often related to substantial damage or foreign matter contamination.

[0219] • Full Black: This defect occurs when the entire seed is black, indicating it is fully spoiled or has been affected by mold or rot.

[0220] • Full Sour: A seed that is entirely sour, typically caused by improper fermentation or excessive moisture, which leads to a highly undesirable taste.

[0221] • Cherry / Pod: Seeds that still have remnants of the outer cherry or pod attached, showing incomplete processing.

[0222] • Mold: A seed that shows visible mold or fungus growth, which can happen due to improper storage or excess moisture.

[0223] • Foreign Matter: This refers to the presence of non-seed materials, such as sticks, stones, or other debris, contaminating the seed batch.

[0224] • Severe Insect: This defect refers to seeds that have been severely damaged by insects, either by holes or other visible signs of infestation.

[0225]

[0160] Secondary Defect Types:

[0226] Secondary defects are less severe but still affect the quality and marketability of the seeds. They may involve partial damage or improper development.

[0227] • Part Black: The seed is partially black, indicating partial spoilage or damage but not as severe as full black.

[0228] • Part Sour: The seed is partly sour, showing that fermentation or drying was inadequate in specific areas of the seed

[0229] • Parchment: The seed still has its parchment layer attached, showing incomplete hulling.

[0230] • Floater: These are seeds that float when placed in water, usually due to being underdeveloped or hollow inside.

[0231] • Immature: The seed did not reach full maturity, often resulting in smaller, undeveloped seeds.

[0232] • Withered: The seed has shriveled or dried up, indicating poor growth or draught or drying processes.

[0233] • Shell: The seed has a thin outer layer that remains attached, which is a sign of incomplete hulling, e.g. due to a genetic defect of the seed.

[0234] • Broken: Seeds that are broken, cut or split, often due to mechanical handling during processing. • Hull / Husk: The presence of the seed’s outer shell or husk that should have been removed during processing.

[0235] • Part Insect: Seeds that show signs of partial insect damage but are not as severely affected as those classified under "severe insect."

[0236]

[0161] Additional Defect Types:

[0237] These defects may not always directly affect flavor but indicate other imperfections or contamination that can lower the seed’s value.

[0238] • Flat / Crushed: Seeds that have been flattened or crushed, usually due to mechanical damage during processing or transportation.

[0239] • Crystallized: This refers to seeds that appear crystallized, often due to over-drying or improper curing.

[0240] • Decolored veined: Seeds that show unusual color changes along the veins, indicating improper drying or processing.

[0241] • Decolored reposado: Seeds with a reposado defect have a faded or dull appearance, typically due to over-drying.

[0242] • Decolored amber or montequilla: Seeds with an amber or "buttery" color, which may indicate over-drying or improper fermentation.

[0243] • Decolored overdried: Seeds that are overly dried, resulting in decoloration, often impacting flavor quality.

[0244] • Wet: Seeds that are too moist, usually caused by improper drying or storage, making them more prone to spoilage or mold.

[0245] • Stick: This refers to the presence of foreign matter such as a stick mixed in with the seeds.

[0246] • Stone: The presence of stones, which are foreign objects that can contaminate a batch of seeds.

[0247] • other: There may be many other defects.

[0248]

[0162] Healthy Seed Categories:

[0249] In addition to defect categories, seeds may also be classified as healthy, meaning no significant defects were detected.

[0250] • Good: This indicates the seed is free from defects and is of high quality.

[0251] • Foxy: A specific category where the seed shows a reddish tint, which may still be acceptable depending on the market.

[0252] • Peaberry: A naturally occurring seed shape where the seed is round rather than flat, considered a desirable or neutral characteristic. • Decolored: This refers to seeds that show some decoloration but not enough to classify them as defective (“decolored” may be considered as an additional defect type or as a healthy seed category, i.e. no actual defect).

[0253] • Wet: A classification for seeds that are not properly dried, indicating potential issues but not necessarily a severe defect (“wet” may be considered as an additional defect type or as a healthy seed category, i.e. no actual defect).

[0254]

[0163] By classifying seeds into these categories, the quality control process ensures that only seeds meeting the required standards proceed for further use, while defective seeds are appropriately identified for rejection or reprocessing. This system helps maintain consistency, ensure product quality, and protect the integrity of the agricultural supply chain

[0255]

[0164] The classifier model may be trained to classify the seed, based on the seed instance, into at least one predefined defect category. For example, the classifier model may output up to two defect categories for a seed.

[0256]

[0165] The classifier model may determine based on each view of the seed instance at least one defect category or up to two (or a predefined number which is more than two) defect categories. The classified defect categories of the seed may comprise a sum of the defect categories determined for the views of the respective seed.

[0257]

[0166] Generally, the classified defect categories of the seed may be determined based on the defect categories determined for the views of the respective seed.

[0258]

[0167] For example, in case a defect category determined for any one of the views of a seed instance is above a predefined threshold, this defect category may be assigned to the seed (i.e. on seed instance level). Optionally, only up to two defect categories for each side may be assigned to the seed, e.g. those having the highest probability. In case more than two defect categories are assigned to the seed, the two defect categories with the highest probabilities may be maintained, e.g. as major defect category and minor defect category.

[0259]

[0168] In case a defect category determined for a first view of a seed instance is a “healthy” category but for any other view of said seed instance an actual defect category is determined, the healthy category is discarded on seed instance level. Accordingly, if a seed shows on one side an actual defect and on another not defect, the seed may anyway be classified as having a defect. Only in case for all views of a seed instance, a “healthy” category is determined with having the highest probability for each view, the seed may be classified as undefected.

[0260]

[0169] The method of steps 100 (i.e. of steps SI to S6) may be repeated (e.g. in a loop 101) for a set of seeds, i.e. for the respective primary images and seed instances of the set of seeds. For this purpose the method 100 (i.e. of steps SI to S6) may be applied to primary image and seed instance of the set of seeds, in order to output a defect category for each seed instance.

[0170] For example, the set of seeds may be a selection of seeds (e.g. 1000 to 3000 seeds) taken out of a larger seed collection (e.g. a harvest volume or a sold volume of seeds). The set of seeds (i.e. the respective primary images and / or seed instances of the set of seeds) may be used to evaluate the quality of the larger seed collection, e g. to detect potential defect categories in the larger seed collection.

[0261]

[0171] Accordingly, in an optional step S7 defect categories may be assigned to the set of seeds (i.e. to the respective primary images and / or seed instances, and optionally thus to the larger seed collection), based on the defect categories outputted by the classifier model (cf. step S6) for each seed of the set seeds.

[0262]

[0172] Since some defect categories are more severe than others (cf. the primary, secondary and additional defect types, as described above), the defect categories may be weighted according to a predefined weighting scheme or metric. For example, the primary, secondary and additional defect types (as described above) may be weighted with different predefined weights, e.g. a primary defect type with a weight of 1, a secondary defect type with a weight of 0.3 or 0.2 and an additional defect type with a weight of 0.1. Accordingly, less severe defect types, such as secondary or additional defect types need to occur more often in the set of seeds (e.g. 10 times more) to be considered as having the same relevance in the set of seeds.

[0263]

[0173] The set of seeds (i.e. the respective primary images and / or seed instances of the set of seeds and optionally also the larger seed collection) may be assigned with one or several defect categories based on the outputted and optionally weighted defect categories for each seed. For example, the sum of all defect categories may be assigned to the set of seed instances (and thus optionally also the larger seed collection), optionally after weighting the defect categories. Alternatively, only the top n defect categories (of all outputted defect categories) may be assigned to the set of seeds, optionally after weighting the defect categories, n being a natural number. The top n defect categories may be determined based on the number of occurrences of particular defect categories, which have been outputted for the seeds, optionally after weighting the defect categories.

[0264]

[0174] Moreover, the classifier model may process the set of seeds (i.e. the respective primary images and / or seed instances of the set of seeds) at least partially in parallel by using multiple parallel processing streams executed by one or more processing units, such as GPUs, to output predicted classes of the different seeds (cf. steps S5 and S6).

[0265]

[0175] This process may be due to the fact that steps SI to S4 are relatively fast, i.e. require less computational resources. For this reason, these steps may be performed successively, i.e. for one seed after the other. However, the classification step (S5 and S6) may be computationally more expensive, as it may involve using a deep learning model. For this reason, the classification step (S5 and S6) may be parallelized, i.e. different seeds may be classified in parallel. For, multiple parallel processing streams executed by one or more processing units, such as GPUs, may be used to classify multiple seeds simultaneously.

[0266]

[0176] In summary, the present disclosure may provide a method for rapid and accurate quality analysis of seeds and grains using computer vision and machine learning algorithms. This method involves high-speed capturing of multi-sided images, preprocessing them, and analyzing various quality parameters. The images are fed into one or more machine learning models trained on expert-labelled data and analyzed by computer vision algorithms. The imaging system, managed by a microcontroller connected to the computing system of the present disclosure, ensures optimal image capture and efficient processing using parallelization of threads.

[0267]

[0177] Fig. 4 shows a schematic exemplary representation of a computing system 400 on which software modules according to the present disclosure are running.

[0268]

[0178] For example, the computing system (i.e. a data processing system) may comprise a processor configured to carry out the method according to the first aspect of the present disclosure. For example, the computing system may further comprise at least one memory storing computer-executable instructions, the computer-executable instructions when executed by the processor cause the computing device and / or its processor to carry out the method according to the present disclosure.

[0269]

[0179] The processor (or processing unit) may be a component of electronic devices that may be responsible for carrying out computational tasks. There may be different types of processing units, each designed for specific purposes. A processing unit may be or may comprise a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), and / or Application-Specific Integrated Circuit (ASIC).

[0270]

[0180] The software modules may comprise computer algorithms and machine-learning based and / or deep-learning based models. The software modules may form a pipeline of modules for processing a primary image 300, i.e. output a defect category based on the primary image.

[0271]

[0181] The computing system 400 may receive and process a primary image 300 taken by a camera system. The computing system may comprise one or several computers, servers or may be at least partially cloud-based.

[0272]

[0182] A segmentation module 410 may be used for at least one of (in particular, in response to receiving the primary image 300, the segmentation module may perform at least one of segmenting the primary image into a plurality of segmented views of the seed, and optionally annotating each view of the plurality of segmented views with a segmentation mask on the primary image.

[0273]

[0183] For this purpose, the segmentation module 410 may comprise at least one of:

[0274] a pre-processing module 411 for at least one of: correcting lens distortion of the primary image, cropping peripheral areas of the primary image, and isolating the views from the background and / or foreground,

[0275] a computer vision algorithm 412, optionally comprising a trained machine-learning based segmentation model, for at least one of: edge detection, contour finding, watershed segmentation to separate each view in the primary image from its background and / or foreground, and recognizing predefined patterns and features in the primary image, a trained deep-learning based segmentation model 413 for segmenting the primary image, optionally pixel by pixel, for distinguishing each seed instance from its background or foreground, and optionally identifying for each seed instance a predefined defect category for segmenting the primary image, optionally pixel by pixel, for distinguishing each seed instance from its background or foreground, and optionally identifying for each seed instance a predefined defect category.

[0276]

[0184] The pre-processing module 411 may be configured for undistortion, cropping, and background removal. Undistorting corrects lens distortions like barrel or pincushion distortion, ensuring accurate representation of the green coffee beans, which is crucial for further processing. Cropping focuses on the relevant portion of the image by removing unnecessary background or peripheral areas, optimizing processing efficiency. Background Removal isolates the green coffee beans from the background using techniques like color thresholding or background subtraction, enhancing the accuracy of subsequent segmentation and analysis.

[0277]

[0185] The segmentation module may isolate individual seeds or grains in images, ensuring an accurate count and analysis. This process uses a combination of computer vision, machine learning (ML), and deep learning (DL) techniques. Computer vision algorithms employ methods like edge detection, contour finding, and watershed segmentation to separate each seed or grain, even in cases of overlap. ML-based segmentation enhances this by using algorithms trained on labeled datasets to recognize specific patterns and features, employing techniques such as support vector machines (SVM) and k-means clustering. DL-based segmentation utilizes convolutional neural networks (CNNs), like U-Net or Mask R-CNN, trained on extensive annotated datasets for pixel-wise segmentation, handling complex images with overlapping seeds and grains. Integrating these approaches ensures each seed or grain is accurately identified and isolated, providing precise segmentation for subsequent processing.

[0186] The segmentation module may further perform e.g. a multi-sided image standardization includes discrimination, evaluating segmented regions across multiple frames to accurately separate each seed or grain for precise counting. Labeling assigns a unique identifier (front (b), back left (a), and back right (c)) to each seed or grain, enabling traceability during multisided analysis. Resizing adjusts image dimensions uniformly, including background reconstruction if needed, ensuring standardized appearance for further analysis.

[0278]

[0187] The segmentation module may further perform e.g. feature extraction including but not limited to Bounding Box determination, Area Calculation based on the number of pixels within the seed’s boundary, Perimeter Measurement the length of the seed and grain's boundary, Major and Minor Axis definition for elliptical fitting, determine the lengths of the major and minor axes of a seed and grain. Image preprocessing involves normalization to standardize pixel values and enhancement techniques like contrast adjustment and noise reduction to improve image quality.

[0279]

[0188] The segmentation module may further perform e.g. size, color and density analysis: Color analysis includes converting images from RGB to perceptually uniform color spaces (HSV or Lab), calculating mean color, analyzing color histograms, extracting color moments (mean, variance, skewness), and measuring color variance. Shape analysis uses edge detection algorithms (Canny or Sobel) to identify boundaries, contour analysis to extract shapes, and geometric features to measure attributes like area, perimeter, aspect ratio, and roundness. Texture analysis employs the Gray-Level Co-occurrence Matrix (GLCM) for contrast, correlation, energy, and homogeneity, Local Binary Patterns (LBP) for intensity comparison, and wavelet transforms for frequency decomposition. Size measurement is performed by converting pixel measurements to real-world units (e.g., millimeters) using a known scale or calibration object in the image. The following relevant features are extracted during size measurement: Length and Width: Measure the major and minor axes of the bean, Area: Compute the area by counting the number of pixels within the bean's boundary and converting it using the calibration factor, Volume Estimation: Based on the 3D information, estimate the volume of the beans using appropriate geometric approximations. Density Estimation: Based on the Volume Estimation and weight of the sample, estimate density of the seeds and grains

[0189] A multi-seed detection algorithm 420 may further identify, among the plurality of segmented views, a set of segmented views relating to the same seed based on the relative positions of the segmented views in the primary image.

[0280]

[0190] Optionally, the multi-seed detection algorithm 420 may be part of the segmentation module 410.

[0191] Moreover, the multi-seed detection algorithm 420 may form a seed instance which shows the segmented views relating to the same seed. The seed instance may comprise at least one of:

[0281] the primary image, wherein the shown views are annotated with respective segmentation masks,

[0282] a single secondary image having a single canvas on which the segmented views are placed adjacent to each other, and

[0283] a set of secondary images, wherein for each view a respective secondary image has a canvas, on which the segmented view is placed

[0284]

[0192] A classifier model 430 may output (e.g. predicts) one or several defect categories in response to the fed seed instance. The classifier model may be configured to perform a multi-label defect classification based on a fed seed instance (in short, the classifier model may be a multi-label classifier model). Accordingly, the output of the classifier model may comprise a multi-label structure. In other words, the model may accept several true answers (i.e. classes or labels) as output. Hence, the classifier model may assign multiple labels to a seed instance, allowing the seed instance to belong to more than one defect category simultaneously.

[0285]

[0193] The classifier model 430 may e.g. employ statistical and machine learning methods to identify anomalies, segment images to isolate individual seeds or grains, and classify defects using a combination of computer vision, machine learning (ML), and deep learning (DL) techniques. During defects analysis, the model extracts relevant features such as categorical labels, which classify defects into predefined categories like fungal infection, mechanical damage, and insect damage. Each image is assigned one or more defect types, providing specific information about the defects present. Severity scores indicate the extent of the defects, ranging from minor to severe, and probability scores express the likelihood of a particular defect type, with higher probabilities indicating greater confidence in the presence of a defect.

[0286]

[0194] The classifier model 430 may comprise a first model 431 which is trained to detect anomalies based on the seed instance. Accordingly, the first model may be trained to distinguish seeds comprising no defect (i.e. “healthy” seeds) from seeds comprising potentially one or several defects (i.e. defected or suspicious seeds). The first model may hence be e.g. a bi-classifier model. For example, a seed instance is classified as defected, i.e. suspicious, in case at least one view potentially shows a defect. The first model may for this purpose classify each view of a seed instance separately, or all views together.

[0195] Furthermore, the classifier model 430 may comprise a second model 432 which is trained to output a defect category of a seed. In particular, the second model may be fed only with seed instances of defected, i.e. suspicious seeds. The second model may be a multi-label classifier model. The second model may for this purpose classify each view of a defected, i.e. suspicious seed instance separately, or all views together.

[0287]

[0196] The second model may comprise a CNN 432a (convolutional neural network) which extracts features of the seed instance, in particular of a segmented view of the seed instance.

[0288]

[0197] Moreover, the classifier model may comprise a classifier neural network 432b (e.g. one or several layers on top of the CNN), which outputs defect categories based on the extracted features. The output layer of the classifier neural network 432b may e.g. comprise a dedicated sigmoid output layer. Accordingly, compared to a conventional classifier model (e.g. a binary or multiclass classifier), the output layer (i.e. last activation function of the model) may be changed from a softmax output layer to a sigmoid output layer. Sigmoid may be suitable for multilabel classification because it allows for independent probabilities for each label, while softmax is designed for exclusive class assignments, such as in multiclass classification.

[0289]

[0198] The first and / or second model may be e.g. a trained deep-leaming model. However, the first model may be e.g. a relatively small and fast model compared to the second model. Accordingly, the speed of the method and / or its computational costs may be reduced by using the first and second models, since only the seed instances of defected seeds are analysed by the second model.

[0290]

[0199] Furthermore, a seed annotation module 440 may, based on the defect categories outputted for each view, assign up to predefined maximum number of defect categories (e.g. up to two) to the seed instance.

[0291]

[0200] The seed annotation module 440 may also be a set of seeds annotation module 440. Accordingly, the pipeline 410, 420, 430 may process, i.e. classify a set of seeds. Based on the defect categories outputted for each seed, the set of seeds annotation module 440 may assign up to predefined maximum number of defect categories (e.g. up to two) to the set of seeds.

[0292]

[0201] At least the classifier model, optionally also the machine-learning model(s) of the segmentation module, may be trained using a training dataset.

[0293]

[0202] For example, a human-graded dataset may be used for developing a robust machine learning model for defect analysis in seeds and grains. This step ensures that the dataset used for training the model may be comprehensive, accurate, and representative of the various types of seeds and grains and defects that can occur. For example, the process includes several detailed sub-steps: • Guideline Development involves creating detailed guidelines for the manual grading process. These guidelines ensure consistency and accuracy across different graders and include Defect Classification that contains clear definitions and examples of each defect type (e.g., decoloration, fungal infection, mechanical damage, insect damage, shriveling), Grading Criteria for evaluating the severity of each defect, including visual indicators and severity levels and Annotation Standards with instructions on how to annotate images, including bounding box labeling, semantic labeling, and instance labeling.

[0294] • Manual Inspection: Qualified agricultural experts manually inspect and grade a large sample of seeds and grains according to the developed guidelines. This involves Visual Inspection when experts examine each seed and grain visually, identifying and categorizing any defects and Severity Assessment when experts assess the severity of identified defects, assigning severity scores based on predefined criteria or industry guideline.

[0295] • Scanning and Storing images of Seeds and Grains to capture High-resolution multi-sided images of the graded seeds and grains using a specialized imaging setup. This setup ensures that images are clear, detailed, and consistent.

[0296] • Image Labeling: The captured images are then annotated by qualified agricultural experts following the guidelines. This involves several types of labeling like Bounding Box Labeling that means drawing rectangles around defective areas to localize defects within the image. Instance Labeling that means distinguishing between individual instances of defects, even if they belong to the same class and Semantic Labeling that means assigning a defect class to each labeled area (e.g., decoloration, fungal infection).

[0297] • Quality Control: To ensure the accuracy and reliability of annotations, a comprehensive quality control process is implemented in three stages. First, annotations undergo review by senior agricultural experts to verify compliance with established guidelines. Second, inter-annotator agreement checks for consistency among annotators to ensure uniformity in annotations. Finally, correction and feedback are provided to annotators, addressing any discrepancies and ensuring high-quality annotations are maintained throughout the process.

[0298]

[0203] The training process may involve feeding the annotated images of seeds and grains into the machine learning model, allowing it to learn the distinguishing various features: edge, position, defects. Advanced techniques such as data augmentation and regularization are employed to enhance the model's performance and generalizability. The training process involves 2 major steps:

[0299] • Architecture Selection: A suitable segmentation model architecture is chosen, often based on deep learning frameworks such as U-Net, Mask R-CNN, or variants of convolutional neural networks (CNNs). These models are adept at learning to segment images pixel by pixel, distinguishing seeds and grains from their background and potentially identifying different types of defects. Classification models like convolutional neural networks (CNNs), support vector machines (SVMs), or ensemble methods are trained on the extracted features. These models learn to classify segmented regions into predefined categories such as healthy grains or various types of defects.

[0300] • Training with Annotated Data: The segmentation model is trained using the annotated dataset. During training, the model iteratively adjusts its parameters based on backpropagation, minimizing the selected loss function to improve segmentation accuracy. Similar to segmentation, the classification model undergoes training using the annotated dataset. It learns to associate extracted features with corresponding labels, optimizing its parameters to maximize classification accuracy. Evaluation metrics such as accuracy, precision, recall, and Fl-score are utilized to measure the model's performance.

[0301]

[0204] The software module pipeline may be e.g. configured for process parallelization. Process parallelization may be applied for maximizing the throughput of the seed and grain analysis pipeline, ensuring efficient, accurate, and rapid quality analysis. By distributing tasks across multiple processors or threads, each step of the process operates simultaneously, significantly enhancing efficiency. This method automates the entire analysis process, reducing the need for manual inspection and significantly increasing throughput. The system's ability to capture and analyze high-quality images of seeds and grains from multiple sides at high speed makes it a valuable tool for the agricultural industry, enhancing quality control processes and ensuring better product standards.

[0302]

[0205] The software module pipeline may further be e g. configured for iterative improvement. Iterative improvement is a critical aspect of the seed and grain analysis pipeline, ensuring continuous enhancement of model performance. This involves continuous monitoring and updating of the machine learning models, allowing for iterative refinements. New annotated data, including images graded by human experts, and advanced techniques such as data augmentation and regularization are regularly incorporated to boost model accuracy and adaptability to various seed and grain varieties. By leveraging process parallelization, the system can efficiently handle high volumes of new data and updates, ensuring rapid, accurate, and automated quality control. This ongoing improvement process enhances the system's ability to capture and analyze high-quality, multi-sided images, further reducing the need for manual inspection and increasing throughput. The result is a highly efficient tool that continually adapts and improves, ensuring better product standards and quality control in the agricultural industry.

[0206] By providing the method and computing system of the present disclosure, an efficient and accurate method and system for the rapid quality evaluation of seeds and grains can be provided. While the image capturing involves synchronization of several mechanical components in order to achieve the consistent quality of images, by utilizing advanced computer vision and machine learning algorithms, the system ensures high throughput and high accuracy in the quality analysis process. The method e.g. achieves to:

[0303] • Automate the quality control process, reducing the need for manual inspection and thereby increasing efficiency.

[0304] • Provide consistent and reliable quality assessments of seeds and grains, essential for ensuring the quality of agricultural products.

[0305] • Facilitate rapid processing and analysis, enabling real-time quality control in high throughput environments.

[0306] • Integrate seamlessly with existing agricultural processing systems to enhance overall productivity and product quality.

[0307]

[0207] Throughout the description, including the claims, the term “comprising a” should be understood as being synonymous with “comprising at least one” unless otherwise stated. In addition, any range set forth in the description, including the claims should be understood as including its end value(s) unless otherwise stated. Specific values for described elements should be understood to be within accepted manufacturing or industry tolerances known to one of skill in the art, and any use of the terms “substantially” and / or “approximately” and / or “generally” should be understood to mean falling within such accepted tolerances.

[0308]

[0208] The terms “record” and “receive” may be used synonymously throughout this disclosure unless denoted differently.

[0309]

[0209] Although the present disclosure herein has been described with reference to particular examples, it is to be understood that these examples are merely illustrative of the principles and applications of the present disclosure.

[0310]

[0210] It is intended that the specification and examples be considered as exemplary only, with a true scope of the disclosure being indicated by the following claims.

[0311]

[0211] A reference herein to a patent document or any other matter identified as prior art, is not to be taken as an admission that the document or other matter was known or that the information it contains was part of the common general knowledge as at the priority date of any of the claims.

Claims

AMENDED CLAIMS received by the International Bureau on 30 April 2026 (30.04.26)CLAIMS1. A computer-implemented method of classifying seeds using a classifier model, the method comprising: receiving a set of primary images, each primary image showing, for at least one seed, views of different sides of the seed; for each primary image, feeding the primary image into a segmentation module and segmenting the primary image into a plurality of segmented views, and creating, from the segmented views, at least one seed instance showing views of different sides of a respective seed; feeding each seed instance into the classifier model, wherein in response the classifier model outputs, for the seed instance and for each of a plurality of defect categories, a probability score indicating a likelihood that the seed exhibits the defect category; for the set of seed instances created from the set of primary images, applying a predefined weighting scheme to the probability scores to obtain, for each defect category, a weighted score; and assigning one or more defect categories to the set of seed instances based on the weighted scores.

2. The method of claim 1, wherein the segmentation module is configured to assign a segmentation mask to each segmented view, and wherein the seed instance comprises at least one segmentation mask.

3. The method of claim 2, wherein the seed instance comprises at least one of:(i) the primary image annotated with the segmentation mask, and(ii) the segmented views as separate secondary images each annotated with a respective segmentation mask.

4. The method of any one of claims 1 to 3, wherein creating the seed instance comprises combining segmented views corresponding to different viewing sides of the seed such that the seed instance represents an entire surface of the seed.

5. The method of claim 4, wherein each segmented view is assigned a type of viewing side selected from a predefined set of viewing sides.

6. The method of any one of claims 1 to 5, wherein the primary image is obtained by an imaging system comprising an imaging sensor and a mirror system configured to capture, in the primary image, a front side, a back-left side and a back-right side of the seed.

377. The method of any one of claims 1 to 6, wherein at least one primary image shows views of different sides of a plurality of seeds, and creating the seed instance comprises: identifying, among the segmented views, a set of segmented views relating to the same single seed by determining a level of a centroid for each segmented view in one dimension of the primary image and adding segmented views to the set whose centroid levels are within a predefined level range; and creating the seed instance from the identified set of segmented views.

8. The method of any one of claims 1 to 7, wherein the classifier model outputs probability scores for a plurality of defect categories simultaneously using a sigmoid output layer.

9. The method of claim 8, wherein the classifier model comprises a first model configured to perform anomaly detection and a second model configured to perform multi-label classification, the method comprising: feeding the seed instance into the first model; and in response to an output of the first model indicating an anomaly, feeding the seed instance into the second model to obtain the probability scores.

10. The method of any one of claims 1 to 9, wherein each defect category is associated with a predefined weight, and applying the predefined weighting scheme comprises, for each defect category, computing the weighted score by combining probability scores output for the defect category with the predefined weight.

11. The method of claim 10, wherein computing the weighted score comprises summing, over the set of seed instances, products of the probability score for the defect category and the predefined weight.

12. The method of any one of claims 1 to 11, wherein feeding seed instances into the classifier model is performed in parallel for a plurality of seed instances.

13. A method of generating training samples for an image segmentation and classification module, the image segmentation and classification module comprising an image segmentation module and an image classifier model, the method comprising: providing a set of primary images, each primary image showing, for at least one seed, views of different sides of the seed; and labelling each view by at least one of: assigning a segmentation mask to the view, assigning a seed identifier to the view which distinguishes the seeds shown in the primary image, assigning a type of viewing side from a predefined number of viewing sides to the view, and38assigning at least one defect category to the view, the defect category indicating a defect of the seed, the defect category optionally further comprising at least one of a severity score indicating the severity of the defect, and a probability score indicating the likelihood of the defect.

14. A method of training an image segmentation and classification module, the image segmentation and classification module comprising an image segmentation module and an image classifier model, the method comprising: providing training samples generated by the method of claim 13; and training the image segmentation and classification module in a supervised manner using the training samples.

15. The method according to any one of claims 1 to 14, further comprising: running the trained image segmentation and classification module by feeding a set of customized primary images into the module; identifying incorrect defect categories among the outputted defect categories; generating a customized training dataset by assigning secondary samples having an incorrect defect category with a correct defect category, the secondary samples being outputted by the segmentation module in response to feeding respective primary images into the segmentation module; and retraining the classifier model using the corrected secondary samples.

16. A computer-implemented method of classifying a seed using a classifier model based on a video stream, the method comprising: receiving the video stream comprising a sequence of image frames, each image frame being a primary image showing at least one seed; for each primary image, feeding the primary image into a segmentation module and segmenting the primary image into a plurality of segmented views; determining a centroid coordinate for each segmented view in one dimension of the primary image; tracking a seed across a plurality of successive primary images by, for each successive primary image, associating segmented views with a tracked seed when their centroid coordinates are within a predefined range relative to centroid coordinates of segmented views previously associated with the tracked seed, thereby accumulating a set of segmented views for the tracked seed; creating a seed instance from the accumulated set of segmented views; feeding the seed instance into the classifier model, wherein in response the classifier model outputs, for each of a plurality of defect categories, a probability score for the seed instance;aggregating the probability scores for the tracked seed across the plurality of successive primary images; and assigning a defect category to the tracked seed based on the aggregated probability scores.

17. The method of claim 16, wherein aggregating comprises summing or averaging, for each defect category, probability scores output for the defect category across the plurality of successive primary images.

18. The method of claim 16 or 17, further comprising applying a predefined weighting scheme to the aggregated probability scores to obtain weighted aggregated scores, and wherein assigning comprises selecting at least one defect category based on the weighted aggregated scores.

19. The method of any one of claims 16 to 18, wherein the predefined range is determined based on a predefined trajectory of the seed in the video stream.

20. The method of any one of claims 16 to 19, wherein creating the seed instance comprises selecting segmented views corresponding to different viewing sides of the seed from the accumulated set of segmented views.

21. The method of any one of claims 16 to 20, wherein the video stream is captured while the seed is transported by a rotating disk or rotating plate along the predefined trajectory, and wherein the centroid coordinate is evaluated along a transport direction of the seed.

22. The method of claim 13, wherein the primary images comprise image frames of a video stream, and the seed identifier distinguishes a seed across a plurality of image frames.

23. A computing system comprising a processor and a memory storing instructions which, when executed by the processor, cause the computing system to carry out the method of any one of claims 1 to 22.

24. A non-transitory computer-readable medium comprising instructions which, when executed by a computing system, cause the computing system to carry out the method of any one of claims 1 to 22.[0001]Statement under Article 19(1) (Rule 46.4)[0002]Statement under Article 19(1)[0003]The Applicant has amended the claims under PCT Article 19 and Rule 46 following receipt of the International Search Report and Written Opinion (date of mailing: 02 March 2026). The amendments replace all claims originally filed with a complete amended set of Claims 1-24.[0004]The amended independent claim directed to classification from primary images (Claim 1) now requires applying a predefined weighting scheme to defect-category probability scores produced for a set of seed instances and assigning one or more defect categories based on the resulting weighted scores. This amendment is intended to further define the claimed decision logic for classifying a set of seed instances and to distinguish the claimed method from approaches that merely output defect classifications for individual seed instances.[0005]In addition, the amended independent claim directed to classification based on a video stream (Claim 16) has been drafted to explicitly introduce the segmentation module and classifier model and to require tracking a seed across successive frames using centroid-coordinate association, aggregating defect-category probability scores across frames, and assigning a defect category based on the aggregated scores. This amendment is intended to more precisely define the temporal aggregation and assignment logic for video-stream processing.[0006]These amendments do not require any amendment to the description or drawings. The description and drawings as filed continue to support the amended claims.

Images