Machine learning based assessment of food quality

Abstract

Described herein are systems and methods for identifying a quality level of food products using image data, such as time-series RGB, hyperspectral, thermal, and / or multispectral images. The method may include receiving image data of food products from an imaging device, performing object detection on the image data to identify a bounding box around each food product, and identifying a quality level of each food product by applying a trained model to the bounding box. The model was trained using image training data of other foods that were annotated based on a previous identification of a first portion of the other food product as having poor quality characteristics and a previous identification of a second portion of the other food product as having good quality characteristics. The other foods and the food products are of the same type. The method also includes determining a quality level score for each food product based on the identified quality level of the food product.

Description

[Technical field]

[0001] Incorporation by Reference [1] This application claims priority to U.S. Provisional Patent Application No. 63 / 295,172, filed December 30, 2021, which is incorporated by reference in its entirety.

[0002] Technical Field [2] This document describes devices, systems, and methods generally relating to identifying food quality, for example, based on image data of the food. [Background technology]

[0003] background [3] Foods such as produce, fruit, and meat can have different quality metrics that can affect their suitability for consumption and value in the supply chain. Many different stakeholders throughout the supply chain have an interest in evaluating the quality metrics of such foods. As an example, the color of a food product can be an indicator of quality that can be used to sort and grade the food product. Different quality metrics can also influence consumer purchasing decisions.

[0004] [4] Foods with quality metrics such as good color properties can be more valuable than foods with poor color or other poor quality metrics. Color and other quality metrics can be used to indicate ripeness, hardness, spoilage, dryness, flavor, sweetness, and sourness characteristics of a food. Any of these characteristics can be valuable throughout the supply chain and in consumer consumption decisions. For example, browning of a food can indicate spoilage. Early identification of color browning can be used to make changes in the supply chain to avoid food being wasted.

[0005] [5] It can be difficult to objectively and quantitatively define high-quality foods based on image data. Relevant stakeholders in the supply chain can observe and compare colors or other characteristics visible in or on foods. However, the human eye can only distinguish color or other characteristic differences in extreme cases or when the color or other characteristic differences are quite obvious. For example, the human eye may be able to distinguish when a lime is yellow, green, or brown, or when a lime is more yellow than other limes, but the human eye may not be able to discern more subtle changes in color. As another example, it is difficult for the human eye to universally rank and compare colors across all possible colors that a particular food product can acquire during its lifespan. Subtle changes in color and other characteristics throughout the lifespan of a food product can represent quality changes in that food. Summary of the Invention [Means for solving the problem]

[0006] overview [6] This document generally describes systems, methods, and techniques for non-invasively assessing the quality of food products (e.g., vegetables, fruits, meats) based on, for example, image data. The image data can be generated and captured by one or more imaging devices, such as cameras, configured to capture images of the food products within the visible light spectrum and / or outside the visible light spectrum. For example, the disclosed technology can capture and use hyperspectral image data captured by a hyperspectral camera. The image data can include images (e.g., RGB, hyperspectral, multispectral, thermal, etc.) and additional metadata. As an illustrative example, the disclosed technology can be used to assess the color of food products to identify food quality such as ripeness, readiness for consumption, mold, spoilage, and / or dryness. The disclosed technology can distinguish color differences, whether obvious or subtle, and define which different colors foods are without supervision. For example, the disclosed technology can distinguish the quality of green apples from yellow apples. A quantitative color score can be inferred for each food product. The image data can also be used to identify many other characteristics and qualities. For example, mold, texture, defects, and dryness can be identified from the image data to determine food quality. Models can be trained using machine learning techniques, such as convolutional neural networks (CNNs), to identify and score such metrics from image data. As a result, food quality can be assessed, especially when the human eye cannot provide an objective, universal quantification or distinguish subtle changes in food such as bruising, infection, aging, ripeness, taste, or other non-visible characteristics of food quality.

[0007] [7] The disclosed technology can provide for generating different metrics that can be used to identify different characteristics of a food product that are indicative of the quality of such a food product. A quality metric can be defined for each food product. A quality metric can also be defined for each type of food product. For example, apples can have different color metrics and corresponding machine learning trained models compared to limes, avocados, and other fruits and produce. Additionally, Granny Smith apples can have different color metrics or other quality metrics compared to Honey Crisp apples. Because the quality of a food product can be determined based on multiple quality metrics, each machine learning trained model corresponding to a quality metric can be run sequentially. Each subsequent model can receive as input the output of a previously run model. Each model can determine a more robust and accurate quality metric score because the models receive the quality metric analysis as input. In some implementations, the machine learning trained models can also be run in parallel.

[0008] [8] A quality score can be determined for each quality metric. An overall quality score for the food product can also be determined based on the aggregation of the individual quality scores. Based on the quality score of the food product, changes can be made early to the supply chain. For example, the quality assessment described throughout this disclosure can be performed once when the food product enters a storage facility. The quality of the food product can be assessed at this point and can be an important indicator of when the food product will be ripe and / or ready for consumption by the consumer. If the quality of the food product is identified as poor (e.g., the food product is currently ripe or past its prime, etc.), the supply chain can be modified to immediately ship the food product to a consumer in the nearest geographic location, discard the food product, or ship the food product to a food processing plant. If the quality of the food product is identified as good (e.g., the food product is beginning to ripen, is ripe, is free of mold or dryness, etc.), the supply chain can be modified to store the food product for some period of time or transport the food product to a consumer in a more distant geographic location. One or more other supply chain changes can be determined based on the preferences of relevant stakeholders throughout the supply chain life cycle.

[0009] [9] Certain embodiments described herein may include a method of identifying a quality level of a food product using image data, the method including receiving, by a computing system, image data of the food product from an imaging device; performing, by the computing system, object detection on the image data to identify a bounding box around each of the food products in the image data; and identifying, by the computing system, a grid structure of the image data based on the bounding box around each of the food products in the image data. Each bounding box may be assigned a grid index in the grid structure, and the grid index may be used to identify the food product in the data store. The method may also include identifying, by the computing system, a quality level of the food product by applying, by the computing system, a plurality of trained models for each of the food products to a bounding box portion of the image data that includes each of the food products. Each of the trained models may be trained using image training data of other foods, the image training data being annotated based on a prior identification of a first portion of the other foods as having a poor quality characteristic and a prior identification of a second portion of the other foods as having a good quality characteristic. The other foods can be of the same food type as the food, and the trained models can include a first trained model trained to identify a first quality feature and a second trained model trained to identify a second quality feature different from the first quality feature. The method can further include determining, by the computing system, for each of the foods a quality level score of the food based on the identified quality level of the food, and storing, by the computing system, for each of the foods in a data store: (i) a bounding box portion of the image data including the food, (ii) a grid index, (iii) the identified quality level of the food, and (iv) the determined quality level score of the food.

[0010]

[10] In some implementations, the embodiments described herein may include one or more of the following features. For example, the method may include transmitting, by the computing system to a user computing device, the quality level scores of the food items in the image data for display on the user computing device.

[0011]

[11] In some implementations, the method may include retrieving, by a computing system, for each of the food products from a data store, a quality level score of the food product; identifying, by the computing system, supply chain information for the food product, which may include an existing supply chain schedule and a destination for the food product; determining, by the computing system, whether to modify the supply chain information for the food product based on the retrieved quality level score; in response to determining to modify the supply chain information, generating, by the computing system, modified supply chain information based on the received quality level score; and transmitting, by the computing system, the modified supply chain information to one or more supply chain participants to implement the modified supply chain information. The modified supply chain information may include one or more of a modified supply chain schedule and a modified destination for the food product. Additionally, the modified supply chain information may include instructions that, when executed by the one or more supply chain participants, may move the food product for outbound shipment to an end consumer that is geographically closest to the location of the food product. In some implementations, the modified supply chain information may include instructions that, when executed by one or more supply chain participants, may cause at least one of: (i) moving the food product for outbound shipment to a food processing plant; (ii) changing the controlled atmospheric conditions surrounding the food product; (iii) changing the aging conditions of the food product; or (iv) changing the cold storage conditions of the food product.

[0012]

[12] As another example, the trained models may each include one or more layers having (i) image training data of other foods and (ii) labels indicating a food quality classification of each of the other items depicted in the image training data. The foods may be at least one of avocado, lime, lemon, apple, berry, and mango. The method may also sometimes include identifying, by the computing system, a type of food depicted in the image data using object recognition, and selecting, by the computing system, one or more of the trained models to apply to a bounding box portion of the image data based on the identified type of food. The selected trained models may be trained to identify quality features of the same type of food. In some implementations, the trained models may be trained using at least one of a convolutional neural network (CNN) and partial least squares (PLS). Additionally, in some implementations, the image data may include at least one of an RGB image, a hyperspectral image, a multispectral image, a thermal image, a nuclear magnetic resonance (NMR) image, and a magnetic resonance image (MRI).

[0013]

[13] In some implementations, by the computing system, identifying, for each of the foods, a quality level of the food may include calibrating a color of the image data to maintain color consistency in the image data; identifying a median Euclidean distance between a color of the food within a bounding box portion of the image data and a reference color of the food based on application of the color analyzer-trained model to the calibrated bounding box portion of the image data, the reference color being associated with a preferred quality level of the food; determining whether the median Euclidean distance exceeds a threshold; and, in response to determining that the median Euclidean distance exceeds the threshold, identifying the quality level of the food as poor quality; and, in response to determining that the median Euclidean distance is less than the threshold, identifying the quality level of the food as good quality.

[0014]

[14] As another example, identifying, by the computing system, for each of the foods, a food quality level can include applying hyperspectral cube processing to a bounding box portion of the image data that includes each of the foods. In some implementations, identifying, by the computing system, for each of the foods, a food quality level can include identifying an extent to which the food has yellowed based on applying an apple yellowing trained model to a bounding box portion of the image data that includes each of the foods, and assigning a food quality level score based on the extent to which the food has yellowed. A score of 0 or greater but below a threshold level can indicate (i) good quality and (ii) no yellowing, and a score of 100 or less but greater than the threshold level can indicate (i) poor quality and (ii) maximum yellowing of the food.

[0015]

[15] In some implementations, by the computing system, identifying a food quality level for each of the foods can include identifying an extent to which the food has yellowed based on applying a lime yellowing trained model to a bounding box portion of the image data including each of the foods, and assigning a food quality level score based on the extent to which the food has yellowed. A score equal to or greater than a first threshold level but less than a second threshold level can indicate (i) good quality and (ii) no yellowing, and a score equal to or less than a third threshold level but greater than the second threshold level can indicate (i) poor quality and (ii) maximum yellowing of the food. In some implementations, applying the lime yellowing trained model can further include applying a Gaussian mixture model and a support vector regressor to a bounding box portion of the image data including each of the foods. In some implementations, applying the apple yellowing trained model can further include applying a Gaussian mixture model and a support vector regressor to a bounding box portion of the image data including each of the foods.

[0016]

[16] In some implementations, by the computing system, for each of the foods, identifying a quality level of the food can include identifying internal quality defects in the food based on applying the internal quality trained model to a bounding box portion of the image data that includes each of the foods, and determining an overall quality level of the food based on the internal quality defects. A score of "good" can indicate that the food has no internal quality defects, a score of "acceptable" can indicate that the food has some internal quality defects, and a score of "poor" can indicate that the food has most internal quality defects. In some implementations, the internal quality trained model can be trained using image training data of other foods that have previously been annotated as having no internal defects, extensive browning, vascular browning, nuclear delamination, internal rot, stem rot, and tissue destruction.

[0017]

[17] As another example, by the computing system, identifying a food quality level for each of the food items can include identifying spoilage and dryness of the food items based on applying the spoilage and dryness trained model to a bounding box portion of the image data that includes each of the food items, and assigning a food quality level score based on the identified spoilage and dryness. The score can be at least one of a binary classification of spoilage and dryness and a numerical value. The numerical value can be a continuous value from 0 to 1 indicating the severity of spoilage and dryness.

[0018]

[18] As yet another example, identifying, by the computing system, for each of the food products, a quality level of the food product can include determining a ripeness and remaining shelf life of the food product based on applying the shelf-life trained model to the image data, and assigning a quality level score to the food product based on the determined ripeness and remaining shelf life. The quality level score can be a numeric value that can indicate the number of days remaining until the food product is ready to eat.

[0019]

[19] In some implementations, by the computing system, identifying, for each of the food products, a food quality level can include identifying a grade of the food product based on applying a produce grade trained model to a bounding box portion of the image data including each of the food products. The grade can be based on USDA specifications, color distribution, shape of the food product, and size of the food product. The method can also include assigning a food quality level score based on the grade. The score can be a string value that can indicate an assessment of at least one of the grade, color distribution, shape, and size of the food product. In some implementations, the string value indicating the assessment of the grade can be at least one of "fine", "prime", "non-fine", "good", "acceptable", and "poor". Further, the string value indicating the assessment of the color distribution can be at least one of "good color" and "poor color". Sometimes, the string value indicating the assessment of the shape can be at least one of "well formed" and "poorly formed". In some implementations, the string value indicating the assessment of the size can be at least one of "good size" and "poor size".

[0020]

[20] As another example, identifying a food quality level for each of the foods by the computing system can include identifying a size of the food based on applying a size-trained model to a bounding box portion of the image data that includes each of the foods, and assigning a food quality level score based on the size. The score can be a numeric value that can indicate whether the food fits into a standard bin size for other foods of the same type as the food.

[0021]

[21] As another example, identifying a food quality level for each of the food items by the computing system can include identifying a shape of the food item based on applying a shape-trained model to a bounding box portion of the image data that includes each of the food items, and assigning a food quality level score based on the shape. The score can be a numerical value that can be indicative of a bending radius of the food item.

[0022]

[22] As yet another example, identifying, by the computing system, for each of the food products, a food quality level can include identifying (i) the presence, (ii) coverage, and (iii) thickness of a ripening coating on the food products based on applying the ripening coating trained model to a bounding box portion of the image data including each of the food products, and assigning a food quality level score based on (i)-(iii). The score can be at least one of a binary inference and a numerical value of pixels in the bounding box portion of the image data including each of the food products that are indicative of the presence, coverage, and thickness of a ripening coating on the food products.

[0023]

[23] In some implementations, by the computing system, identifying a food quality level for each of the foods can include identifying a dry matter content of the food based on applying the dry matter trained model to a bounding box portion of the image data that includes each of the foods, and assigning a food quality level score based on the dry matter content. The score can be a numeric value that can indicate a quantity of the dry matter content of the food.

[0024]

[24] As another example, identifying a food quality level for each of the foods by the computing system can include identifying a hardness of the food based on applying a hardness trained model to a bounding box portion of the image data including each of the foods and assigning a food quality level score, which can be a numeric value, based on the hardness. In some implementations, identifying a food quality level for each of the foods by the computing system can include identifying a sugar content level of the food based on applying a Brix trained model to a bounding box portion of the image data including each of the foods and assigning a food quality level score, which can be a numeric value that can indicate the sugar content in Brix of the food.

[0025]

[25] In some implementations, by the computing system, for each of the foods, identifying a food quality level may include identifying a nutritional content of target compounds in the food based on applying a nutrient-trained model to a bounding box portion of the image data that includes each of the foods, and assigning a food quality level score based on the nutritional content of the target compounds, where the score may include a list of concentrations of the nutritional content of the target compounds in the food.

[0026]

[26] In some implementations, by the computing system, identifying, for each of the food products, a food quality level can include identifying a product label on the food product based on applying a product identifier trained model to a bounding box portion of the image data that includes each of the food products, decode at least one of the text on the product label and the product identifier using optical character recognition (OCR), and assigning a food quality level score based on the decoded product label. The score can be a string value that includes at least one of the decoded text and the product identifier.

[0027]

[27] As another example, identifying, by the computing system, for each of the foods, a food quality level can include identifying an astringency of the food based on applying an astringency trained model to a bounding box portion of the image data including each of the foods, and assigning a food quality level score based on the astringency. In some implementations, identifying, by the computing system, for each of the foods, a food quality level can also include identifying a sourness level of the food based on applying an acidity trained model to a bounding box portion of the image data including each of the foods, and assigning a food quality level score based on the sourness level of the food, where the score can be a numerical value that can be indicative of the sourness level of the food.

[0028]

[28] In some implementations, by the computing system, for each of the foods, identifying a quality level of the food may include calibrating colors of the image data to maintain color consistency in the image data; obtaining a color sample of each food within a calibrated bounding box of the image data; mapping the color samples to the multidimensional color space; identifying a direction of maximum color change across all of the color samples; predicting a location of maximum color change along the identified direction for the food, where the location of maximum color change is a one-dimensional quality metric for the food; and identifying a quality level of the food as good quality based on a determination that the one-dimensional quality metric of the food is within a threshold quality range.

[0029]

[29] In some implementations, the modified supply chain information includes instructions that, when executed by one or more supply chain participants, move the food product for outbound shipment to an end consumer that is geographically closest to the location of the food product. As another example, the quality level score of the food product can be an overall quality metric for the food product, and the overall quality metric can be identified by accessing, from a data store, (i) rule-based mappings of distinct ranges of score values ​​corresponding to identifications of first and second quality characteristics to (ii) enumerated food quality categories, the enumerated food quality categories including at least one of sellable, unsellable, edible, inedible, good quality, poor quality, and acceptable quality, for each of the first and second quality characteristics, iteratively determining whether each rule-based mapping is satisfied, identifying an enumerated food quality category for the food product based on each of the rule-based mappings being satisfied, and assigning the identified category to the food product as the overall quality metric for the food product.

[0030]

[30] In some implementations, by the computing system, for each of the food products, identifying a food quality level can include assigning a binary value indicative of whether wrinkles are present or absent on a surface of the food represented by the bounding box portion based on applying a wrinkle analyzer model to a bounding box portion of the image data including each of the food products; identifying a percentage of wrinkle coverage on the surface of the food represented by the bounding box portion of the image data based on (i) summing a quantity of the bounding box portions that are assigned a binary value indicative of the presence of wrinkles on the surface of the food represented by the corresponding bounding box portion and (ii) dividing the sum by a total quantity of the bounding box portion of the image data; and assigning a food quality level score based on the identified percentage of wrinkle coverage meeting a threshold wrinkle criterion, the score being a numeric value indicative of whether wrinkles are present or absent on the surface of the food product. As another example, by the computing system, identifying a quality level of the food for each of the foods can include identifying a median color value of the food based on applying the calyx browning trained model to a bounding box portion of the image data including each of the foods, and assigning a quality level score of the food based on the median color value of the food. The quality level score can include assigning a string value of (i) "good" based on the median color value being less than a first threshold color range, (ii) "acceptable" based on the median color value being greater than the first threshold color range and less than a second threshold color range, or (iii) "poor" based on the median color value being greater than the second threshold color range.

[0031]

[31] In some implementations, identifying a food quality level for each of the foods by the computing system may include identifying a color value of the food based on applying a banana stage analyzer trained model to a bounding box portion of the image data including each of the foods, and assigning a food quality level score based on mapping the food color value to expected threshold color values ​​for a plurality of ripeness stages. The quality level score may be a string indicating a current ripeness stage of the food. In some implementations, identifying a food quality level for each of the foods by the computing system may include extracting a stem of the food based on applying an object detection model to a bounding box portion of the image data including each of the foods, identifying a median color value of the stem of the food based on applying a cherry stem color trained model to the extracted stem of the food, and assigning a food quality level score based on the median color value of the stem of the food. The quality level score may be assigned (i) a string value of “good” based on the median color value being within a first threshold color range, (ii) a string value of “acceptable” based on the median color value being within a second threshold color range that does not include the first threshold color range, and (iii) a string value of “poor” based on the median color value being within a third threshold color range that does not include at least the first threshold color range.

[0032]

[32] One or more embodiments described herein may include a system for identifying a quality level of a food product using image data, the system including one or more imaging devices capable of measuring image data of food products of the same food product type, and at least one computing system capable of performing the methods and additional features described above.

[0033]

[33] One or more embodiments described herein may also include a system for identifying a quality level of a food product using image data, the system including a photo box and at least one computing system. The photo box may have first, second, third, and fourth walls and a ceiling. The photo box may include an opening in the first wall capable of receiving a shallow box containing food products of the same food product type, a flap that may cover the opening and block ambient light from entering the photo box, at least one light that may be inside the photo box that may provide consistent lighting to illuminate the shallow box containing the food products, and one or more imaging devices that are attached to the ceiling of the photo box and capable of capturing image data of the food products on the shallow box. The at least one computing device may perform the method.

[0034]

[34] In some implementations, the embodiments described herein may include one or more of the above and following features. For example, the at least one light may be an LED light. The photo box may further include a camera rig that may extend across the ceiling of the photo box. The camera rig may include three tubes connected at a T-connector. A first tube may extend from the T-connector to a first wall, a second tube may extend from the T-connector to a second wall, the second wall facing the first wall, and a third tube may extend from the T-connector to a third wall, the third tube being perpendicular to the first and second tubes. In some implementations, one or more imaging devices may be mounted on the camera rig along at least one of the three tubes near the T-connector. Additionally, the one or more imaging devices may be at least one of an RGB camera, a hyperspectral imaging device, a thermal imager, an MRI scanning device, and an NMR imaging device.

[0035]

[35] The disclosed technology may provide one or more of the following advantages. For example, machine learning trained models may be used to more accurately identify food quality from subtle differences found in image data. The human eye may be prone to errors in observing subtle changes in the appearance of food and may not be able to detect invisible features of food. For example, the human eye may not be able to notice a slight discoloration on a part of an apple, which may indicate an early sign of spoilage or other poor quality characteristics. Additionally, human workers must be trained to quantitatively visually measure the color, quality, size, and shape of food. This may be a tedious and time-consuming process, and may be subject to human bias. The disclosed technology provides automatic and accurate detection of different quality characteristics of food from high-quality labeled and / or unlabeled image datasets. The disclosed technology provides analysis of food quality beyond just the visible spectrum and color, all of which may be difficult or impossible to observe and analyze with the human eye. Thus, the disclosed technology may provide deeper analysis, increased efficiency, and reduced human error that may result from observing visible characteristics of food.

[0036]

[36] As another example, the disclosed technology can be used to make appropriate supply chain changes early enough in the supply chain lifecycle to reduce food-based waste. Food quality can be assessed at any time throughout the supply chain. For example, quality can be assessed before the commodity is shipped from the farm to the storage facility. Quality can also be assessed when the food arrives at the storage facility. In some implementations, quality can even be assessed when it is displayed in a supermarket and offered to consumers. If food quality is assessed early in the supply chain lifecycle, foods can be better sorted based on the identification and / or estimated quality of such foods. For example, foods identified as good quality upon entering a storage facility can be stored in that facility for a longer period of time than foods identified as poor quality. Improved decision-making can also be made regarding how and when to process or otherwise mature foods. For example, foods identified as beginning to mature during transportation from the storage facility to the store can be treated with a product intended to accelerate the maturation process. As another example, if one or more foods are determined to not meet a quality level threshold, the disclosed technology can determine that an appropriate supply chain change is to begin applying an antimicrobial treatment to the foods. A consumption time frame may also be specified, which may affect when the food is delivered to the supermarket and how long the food remains in storage.

[0037]

[37] Similarly, quality metric scores determined using the disclosed technology can be advantageous for retailers to identify food return on investment (ROI). The quality metric scores can be used by retailers to determine which foods to order, which foods to display to customers, and / or how to price different foods. Thus, the disclosed technology can be used to monitor food quality throughout the supply chain.

[0038]

[38] As described throughout, the disclosed techniques can generate more robust quality assessments of foods. Different models can be generated and trained using machine learning techniques and high-quality labeled training datasets to identify and score different quality features associated with different foods. Thus, the models can be trained to identify features that may otherwise be difficult for a human worker to observe and associate with the quality of the food. One or more models can be selected for each food and run sequentially such that each model can generate a quality assessment that builds on the quality assessment of the previously run model. As an example, a first model run sequentially can determine a color score for a particular food. The color score can then be used as an input to a second model run sequentially that determines a shelf-life prediction score. A shelf-life prediction score can then be determined based on the color score. Additional models can also be run sequentially. As an illustrative example, color analysis can also be run sequentially. First, a model can be run that identifies all limes that are spoiled and / or dried in the image data. A second model can then run a color analysis on limes that are not spoiled and / or dried. Finally, for these limes, color can be the determining factor for whether or not they are of sufficiently high quality. As another illustrative example, avocado ripeness can be identified using multiple models run in sequence. First, a model can be run to detect stem wither and other defects such as vascular browning. Then, a second model can be run to classify the avocado as good or bad based on the detected stem wither and other defects from the first model. Such robust quality assessments can be advantageous to more accurately monitor food quality and modify the supply chain.

[0039]

[39] As yet another example, the disclosed techniques provide a non-destructive assessment of food quality. Because a model is trained to analyze the quality characteristics of food from image data, a human may not need to perform destructive techniques, such as drilling into the skin or flesh of the food, to identify the quality. As a result, food delivered to an end consumer can be tested and assessed for quality without actually reducing the quality of such food. Thus, a model is trained to extract quality features from the image data of the food, instead of a human having to destroy or otherwise alter the food before it is delivered to the end consumer. A higher quality food can be delivered to the end consumer, and no food may be wasted when the disclosed techniques are used to assess food quality.

[0040]

[40] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features and advantages will be apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0041] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1A]

[41] A conceptual diagram of identifying food quality based on image data. [Figure 1B]

[42] Conceptual diagram of the generation of a model to identify different quality characteristics of food. [Figure 1C]

[43] FIG. 1 is a diagram of an example system for assessing the quality of one or more food products based on image data. [Diagram 2]

[44] A flowchart of a process for identifying food quality based on image data. [Diagram 3]

[45] Flowchart of the process for generating models to identify different quality characteristics of foods. [Figure 4A]

[46] A flowchart of the process of using the model to identify food quality during run-time. [Figure 4B]

[46] A flowchart of the process of using the model to identify food quality during run-time. [Diagram 5]

[47] Figure 1 is another flowchart of a process for assessing the quality of one or more food products. [Figure 6]

[48] ​​FIG. 1 is a system diagram of an example of components used herein to identify food quality based on the wooden foot technique. [Figure 7]

[49] A conceptual diagram of an example photo box used to identify food quality based on image data. [Figure 8A]

[50] shows lime color analysis using image data. [Figure 8B]

[50] shows lime color analysis using image data. [Figure 9]

[51] show produce color analysis using image data. [Figure 10]

[52] presents size distribution analysis using image data. [Figure 11]

[53] present a hardness analysis using hyperspectral imagery (HSI) data. [Figure 12]

[54] shows an example of avocado firmness prediction using HSI analysis. [Figure 13]

[55] provide another example of avocado firmness prediction using HSI analysis. [Figure 14]

[56] show pixel-by-pixel avocado firmness predictions for HSI data. [Figure 15]

[57] present an axis wilt analysis using HSI data. [Figure 16]

[58] presents an example of aging analysis using HSI data. [Figure 17]

[59] shows an example of output from an internal quality analysis of an avocado. [Figure 18]

[60] FIG. 1 is a block diagram of system components that can be used to implement a system for assessing the quality of one or more food products. [Figure 19]

[61] shows an example of mango wrinkle analysis using image data. [Figure 20]

[62] An example of strawberry calyx yellowing analysis using image data is shown. [Figure 21]

[63] An example of banana stage analysis using image data is given. [Figure 22A]

[64] An example of cherry stem color analysis using image data is shown. [Figure 22B]

[64] An example of cherry stem color analysis using image data is shown. [Diagram 23]

[65] A flowchart of the process for identifying the overall quality metrics of a food product. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0042] Detailed Description

[66] The present disclosure relates to systems, methods, and computer programs for assessing the quality of one or more food products from image data. The disclosed technology can provide for obtaining image data of one or more food products and determining a quality score for each food product depicted in the image data based on evaluating the image data. Models can be trained using machine learning techniques to process the image data and determine quality scores for different quality metrics. As described herein, the different quality metrics can be modeled and determined based on the food product and / or food type. Models can be dynamically selected based on the type of food product depicted in the image data and then run sequentially to create an accurate quality assessment of such food product. Additionally, the disclosed technology can provide for modification of one or more supply chain operations based on the quality assessment of the food product in an effort to mitigate losses that may result from food products having quality levels that fail to meet certain thresholds.

[0043]

[67] Referring to the figures, FIG. 1A is a conceptual diagram of identifying food quality based on image data. A computer system 150, an imaging device 160, and a user device 170 can communicate (e.g., wired and / or wireless) via a network 180. The computer system 150 can be configured to assess the quality of an imaged food product, such as a produce product, as described throughout this disclosure (e.g., with reference to FIG. 1C, FIG. 6). The imaging device 160 can include an image sensor 106 and at least one light source 106A (e.g., see FIG. 1C). The imaging device 160 can be located within a storage facility or anywhere else along the supply chain. For example, the imaging device 160 can be a handheld device, such as a mobile phone or tablet, that a human user can use to capture an image of the food product. As shown in FIG. 1A, the imaging device 160 can be located above a conveyor belt 104 within the storage facility. See FIG. 7 for an alternative imaging device 160 configuration.

[0044]

[68] The imaging device 160 may be configured to continuously capture (A) image data of the produce 102A-N as the produce 102A-N moves along the conveyor belt 104 to one or more storage locations or other destinations within the storage facility. The produce 102A-N may be fruits, such as avocados, or other food items that may enter the storage facility. For example, the produce 102A-N may be received at the storage facility from a shipping vendor and placed on the conveyor belt 104. The produce 102A-N may be in a case, in a container, on a pallet, or placed directly on the conveyor belt 104. In some implementations, the produce 102A-N may be stationary in the storage facility or not moving on the conveyor belt 104, as will be further described with reference to FIG. 7. For example, some of the produce 102A-N may be randomly sampled and placed inside a photo box (e.g., see photo box 700 of FIG. 7). A camera located within the photo box can capture image data of the products 102A-N, and the image data can then be analyzed by the computer system 150 to obtain an estimate of the quality distribution of the batch of products 102A-N.

[0045]

[69] The imaging device 160 may transmit the image data to the computer system 150 (B). The computer system 150 may then apply one or more models to the image data to identify characteristics of the produce 102A-N (C). The characteristics may be indicative of a quality of the produce 102A-N. As described herein, the models may be trained using machine learning techniques to identify specific characteristics of different food products. The computer system 150 may select one or more models to apply in (C) based on the type of produce 102A-N identified in the image data. Additionally, as described throughout this disclosure, the computer system 150 may apply a model to each of the produce 102A-N identified in the image data.

[0046]

[70] The computer system 150 may then determine a quality metric score for each of the identified features for each of the products 102A-N in the image data (D). In some implementations, the computer system 150 may determine an aggregate quality score for each of the products 102A-N based on the quality metric scores determined for each of the identified features. The computer system 150 may transmit the quality metric scores to the user device 170 (E). For example, the computer system 150 may transmit each quality metric score for each of the products 102A-N in the image data. The computer system 150 may transmit only some of the quality metric scores for each of the products 102A-N. The computer system 150 may transmit quality metric scores for only some of the products 102A-N. In some implementations, the computer system 150 may transmit only the aggregate quality metric score for each of the products 102A-N.

[0047]

[71] In some implementations, the computer system 150 may also transmit the quality metric scores to a database for storage. The quality metric scores may be stored along with other past measurements and additional metadata associated with each produce 102A-N. This stored information may be used in a feedback loop for the continued refinement and training of machine learning models used to perform the techniques described herein. For example, the previously determined quality metric scores may be used to refine and / or train one or more higher level models to identify quality metrics that depend on seasonality, variety, size, country of origin, and other factors.

[0048]

[72] The user device 170 may output the quality metrics and corresponding scores (F). The user device 170 may be a mobile device, smartphone, tablet, laptop, or other computer that may be used by relevant stakeholders in the supply chain. The stakeholders may view the quality metrics and scores for each product 102A-N to understand or analyze the return on investment (ROI) of the products 102A-N. The output metrics and scores may also be used by the stakeholders to monitor the quality of the products 102A-N over time and, optionally, make one or more supply chain changes based on the current and / or expected quality of the products 102A-N.

[0049]

[73] In some implementations, a stakeholder can provide user input at user device 170 indicating a selection of which features or quality metrics to score for a particular product 102A-N. This input is received by computer system 150 and can be used to select a model to apply to the image data in (C). In some implementations, a stakeholder can also provide input at user device 170 indicating a selection of which quality metric scores to receive at user device 170. Thus, computer system 150 can transmit only the user-desired quality metrics in (E).

[0050]

[74] Optionally, the user device 170 may determine one or more supply chain changes based on the output quality metrics and quality scores (G). The user device 170 may automatically determine or otherwise recommend one or more supply chain changes for the products 102A-N based on the corresponding quality scores of the products 102A-N. In some implementations, the computer system 150 may determine or otherwise recommend supply chain changes and transmit those recommendations to the user device 170. A stakeholder at the user device 170 may optionally implement, modify, or reject any of the recommended supply chain changes. In some implementations, the stakeholder may also review the output quality metrics and scores and determine which supply chain changes to implement.

[0051]

[75] FIG. 1B is a conceptual diagram of the generation of a model identifying different quality characteristics of a food product. As described throughout this disclosure, models can be generated for different foods, different food types, and different characteristics associated with a particular food product. Thus, as shown, computer system 150 can receive and generate image data 190. Image data 190 can include digital RGB images, hyperspectral images, and / or multispectral images showing a particular product, a product type, different products, different product types, a product, and / or a batch of products. Produce image data 190 can include images, tables, and / or other data of a particular product having some particular characteristic to be modeled, such as spoilage and drying, and images of the same type of product without the particular characteristic to be modeled. For example, produce image data 190 can include tables stored in a data store that include features extracted from images of the product (e.g., spoilage, drying, probabilities determining shelf life). In some implementations, produce image data 190 can include images of the exterior of the product and / or the interior of the product. In some implementations, the produce image data 190 may include images of a particular produce at different stages of ripeness and between stages of ripeness. The produce image data 190 may be a robust collection of training data that shows multiple different characteristics that may be present and / or occur in a particular produce throughout the life of the produce. The produce image data 190 may also be a collection of images of the same produce from different angles such that the entire produce may be fully analyzed using the techniques described herein. Additionally, in some implementations, the produce image data 190 may include labels for the characteristics, conditions, and / or qualities of the produce. Additionally, in some implementations, such characteristics, conditions, and / or qualities of the produce may be learned using the produce image data 190 that does not include labels.

[0052]

[76] The computer system 150 can receive image data 190 (A). The image data 190 can be received from one or more imaging devices, such as imaging device 160 (see, e.g., FIG. 1A, FIG. 1C) described throughout this disclosure. The computer system 150 can identify produce characteristics from the image data (B). The computer system 150 can identify characteristics indicative of produce quality. For example, the computer system 150 can identify produce color, spoilage, mold, different types of characteristics, blemishes, etc. from the RGB image data. The computer system 150 can also identify other characteristics, such as hardness and dry matter content, from the hyperspectral image data. The identified characteristics can be labeled (see, e.g., FIG. 3) as further described throughout this disclosure.

[0053]

[77] The computer system 150 can then generate a machine learning model of the identified features (C). In some implementations, the model can be generated and / or trained by one or more other computing systems, computers, networks of devices, and / or cloud-based services. The model can be trained, for example, by a remote computer system, stored in a data store, and accessible and executed by the computer system 150. The model can be generated using machine learning techniques, including but not limited to CNN. One or more other machine learning techniques may be used to generate and train the model. The computer system 150 can generate a model for each of the identified and labeled features. Thus, each model can be trained during runtime use to identify a particular feature from image data of the same type of produce (see, for example, Figures 3, 6). Each model can also be trained to score the quality of the produce based on the identified features. As an example, one model can be trained to identify and score a particular change in color of an apple. Another example model can be trained to identify and score texture changes in the skin of an avocado. Another illustrative model (e.g., a computer vision model) can be trained to identify and score blemishes in one or more different types of food products. Another illustrative model can be trained to identify and score hardness in one or more different types of food products. As another example, a model can be trained to identify and score dry matter content. Another illustrative model can be trained to identify hardness, blemishes, and / or dry matter content based on analysis of the HSI.

[0054]

[78] Another illustrative model can be trained to identify produce quality based on the temperature of the imaged produce. For example, thermal images can be used to assess the temperature of the produce. Some produce can spoil if the temperature is outside the acceptable temperature range for that produce. Additionally, thermal images can be used to verify that the produce has properly dried after application of a shelf-life extension coating solution. For optimal, favorable, or beneficial performance of a shelf-life extension coating solution, the fruit should be completely dry after the application process. Additional models can be trained to score features identified by the models described above (e.g., see FIG. 1C, FIG. 6). The generated models can then be output by the computer system 150 (D). During run-time, one or more of the models can be applied to the image data to identify and score features indicative of the quality of the imaged produce.

[0055]

[79] Figure 1C is a diagram of an example system 100 for assessing the quality of one or more food products based on image data. System 100 can include an image sensor 106, an extraction engine 110, a food detection engine 120, a quality assessment engine 130, and a quality evaluation engine 140. In this disclosure, an "engine" can include one or more software modules, one or more hardware modules, or any combination thereof.

[0056]

[80] The image sensor 106 can be used to generate image data 108 representative of attributes of the food items 102A-N, where N is a positive integer greater than 0 and represents the number of food items 102 on the conveyor belt 104. In the example of FIG. 1C, the image sensor 106 can be positioned to allow the image sensor 106 to capture image data 108 representative of one or more images of the food items 102A-N as the food items 102A-N progress along the conveyor belt 104. In some implementations, the sensor 106 can include one or more hyperspectral sensors configured to capture hyperspectral data representative of characteristics of the food items 102A-N. In such implementations, each pixel of the hyperspectral image can correspond to a spectrum of infrared or ultraviolet light associated with a corresponding food item imaged by a camera equipped with one or more sensors operating in the corresponding spectral range. The spectrum of visible light can be used to reconstruct an RGB image of the food items 102A-N. In some implementations, the sensor 106 can be a low resolution digital camera (e.g., 5M or less), a high resolution digital camera (e.g., 5MP or more), or any other type of sensor capable of capturing image data 108. The sensor 106 can cover a spectral range of 300 nm to 1700 nm. One or more other sensors, such as sensors in a hyperspectral camera, can cover wavelengths ranging from 300 nm to 2500 nm. One or more other cameras may also be used, including, but not limited to, an ultraviolet capture camera.

[0057]

[81] In some implementations, the sensor 106 may include multiple sensors positioned at multiple angles relative to the food items 102A-N. For example, the sensor 106 may include a first camera and at least one additional second camera, each capturing image data 108 of the food items 102A-N from a different angle. In such a configuration, the one or more additional cameras may be used to generate image data 108 based on light at wavelengths different or in addition to those captured by the first camera. In general, any set of wavelengths of light may be acquired by the sensor 106.

[0058]

[82] Each particular one of the one or more cameras can be configured to detect different or additional wavelengths of light in a number of different ways. For example, in some implementations, different sensors can be used in different cameras to detect different or additional wavelengths of light. Alternatively or additionally, each of the one or more cameras can be positioned at different heights, angles, etc. relative to one another to attempt to capture different wavelengths of light. In some implementations, one or more cameras can be positioned to at least partially capture a portion of the food item 102A-N that may be obscured from the view of a first camera.

[0059]

[83] In some implementations, one or more light sources 106A can be used to illuminate the food items 102A-N so that the image sensor 106 can capture clear image data 108 of the food items 102A-N. The light source 106A can include one or more light sources, each generating the same or different electromagnetic radiation. In this example, the light source 106A is shown fixed to the image sensor 106. In some implementations, the light source 106A can be located at one or more locations proximate the image sensor 106 to illuminate the food items 102A-N before and / or during capture of the image data 108. In some implementations, the one or more light sources 106A can be selected based on the frequency of the electromagnetic radiation output. For example, in some implementations, the light source 106A can be a halogen light source. Alternatively or additionally, the one or more light sources 106A can be a diode or a series of broadband light emitting diodes (LEDs) that can be used to provide light across the visible wavelength spectrum, the near infrared wavelength spectrum, the electromagnetic spectrum, or any other spectrum. In general, any light source can be used to provide any type of light to the image sensor 106 .

[0060]

[84] In some implementations, one or more light sources 106A or a control unit for one or more light sources 106A can be communicatively coupled to the image sensor 106 or a control unit for the image sensor 106. For example, the image sensor 106 or a control unit for the image sensor 106 can send a signal to the one or more light sources 106A or a control unit for the one or more light sources 106A that causes the light source 106A to illuminate the food item 102A-N with a particular power and / or with one or more particular wavelengths of light at a particular moment in time. In some implementations, the particular moment in time can be a predetermined amount of time before or during the capture of the image data 108.

[0061]

[85] Image data 108 generated by the image sensor 106 can be provided as input to the extraction engine 110. The image data 108 can include one or more images of the food products 102A-N. Such images can also include one or more HSIs. In some implementations, the image sensor 106 can provide the image data 108 directly to the extraction engine 110. The extraction engine 110 can then process the image data 108. In some implementations, the image sensor 106 can store the image data 108 in a data store and / or memory device. The extraction engine 110 can then access the data store and / or memory device to retrieve and process the image data 108.

[0062]

[86] The extraction engine 110 can acquire / receive image data 108. As shown in FIG. 1C, the image data 108 can be one image of a plurality of food products 102A-N. Thus, the image data 108 can be composed of a plurality of images 108A-N, each image 108A-N corresponding to a food product 102A-N. The image data 108A-N can include a depiction of the food product 102A-N. The image data 108A-N can also include a portion of the surrounding environment of the food product 102A-N, such as the conveyor belt 104 or a portion of a food processing facility (e.g., the photo box 700 depicted in FIG. 7). The extraction engine 110 can process the acquired image data 108 as described throughout this disclosure and extract a portion 112 of the image data 108. As shown, the extracted portion 112 can include the food product 102A-N without the surrounding environment. In some implementations, the extracted portion 112 can include only one of the food products 102A-N.

[0063]

[87] The extracted image portion 112 of the image data 108 (referred to herein as extracted image 112) may be provided as an input to the food identification engine 120. In some implementations, the extraction engine 110 may provide the extracted image 112 directly to the food identification engine 120. In some implementations, the extraction engine 110 may store the extracted image 112 in a memory device, in which case the food identification engine 120 can access the memory device.

[0064]

[88] The food identification engine 120 may use one or more object recognition algorithms and techniques to recognize portions of the extracted image 112 that correspond to the foods 102A-N. As an illustrative example, the foods 102A-N may be avocados. The food identification engine 120 may be trained on multiple images of avocados to determine whether one or more avocados are shown in the extracted image 112 and which regions of the extracted image 112 contain avocados. Thus, the food identification engine 120 may be trained to determine a bounding box around each of the foods 102A-N in the extracted image 112. The engine 120 may also be trained to generate output data in the form of annotated images 122 of the foods 102A-N. In generating the annotated images 122, the engine 120 may be configured to annotate or otherwise index each bounding box 122A-N representing each of the foods 102A-N. As described throughout this disclosure, a machine learning trained model may then be applied to each bounding box region 122A-N to determine a quality metric score for each food item 102A-N. As described throughout this disclosure, the system 100 may be used to assess the quality of different types of food items, including, but not limited to, citrus fruits, mangoes, apples, berries, stone fruits, tomatoes, meats, and / or vegetables.

[0065]

[89] In some implementations, the annotated image 122 may include a coordinate system for annotating or indexing the location of each of the food items 102A-N. Numeric values, such as x and y values ​​in an x ​​and y coordinate system, may be used to represent the location of the food items 102A-N in the annotated image 122. Subsequent processing steps may identify the outer boundary of each food item 108A-N using the numerical values ​​representing the location of the food items 102A-N.

[0066]

[90] The annotated images 122 generated by the food identification engine 120 may be provided as input to the quality assessment system 130. In some implementations, the food identification engine 120 may provide the annotated images 122 directly to the quality assessment system 130. In other implementations, the food identification engine 120 may store the annotated images in a memory device, in which case the quality assessment engine 130 can access the memory device to retrieve and process the annotated images 122.

[0067]

[91] The quality assessment engine 130 can be configured to determine a quality metric score for each of the food products 108A-N depicted in the annotated image 122. As described throughout this disclosure, the engine 130 can use one or more machine learning trained models to determine the quality metric score. Each model can be trained to identify different features indicative of the quality of the food products 108A-N. For example, each model can be executed by a different quality assessment engine 132A-N. Each quality assessment engine 132A-N can be configured to perform a particular quality assessment operation on each annotated image 122 of the food products 108A-N. The annotated images 122 can be processed sequentially by the engines 132A-N. This can be advantageous so that each engine 132A-N can make a more robust and accurate quality assessment of the food products 108A-N based on the quality assessment made by the previous engine 132A-N. In some implementations, the annotated images 122 can also be processed in parallel by the engines 132A-N.

[0068]

[92] As described throughout this disclosure, the quality assessment engines 132A-N may each perform different quality assessment operations. In some implementations, all of the engines 132A-N may be run sequentially. In some implementations, less than all of the engines 132A-N may be run sequentially. For example, a particular set of engines 132A-N may be selected based on the type of food items 108A-N in the annotated image 122. Additionally, in some implementations, one or more of the engines 132A-N may be selected for execution, while other of the engines 132A-N may not be executed based on the quality assessments made by some of the engines 132A-N. For example, if engine 132A determines that the quality of the apple is poor because the apple has browning, then engine 132B may not need to be executed - since the quality of the apple has already been identified as poor. In some implementations, one or more of the engines 132A-N may be executed in parallel. Parallel execution can be advantageous for reducing the amount of time required to process annotated images 122 and perform multiple quality assessments of the foods depicted therein.

[0069]

[93] In some implementations, the set of quality assessment engines 132A-N can be configured based on the type of food product 108A-N being analyzed. In some implementations, the set of quality assessment engines 132A-N can be configured based on the business practices of the business entity implementation system 100. In some implementations, the set of quality assessment engines 132A-N can be dynamically selected by an end user using a computing device (e.g., see user computing device 170 of FIG. 1A) based on quality preferences that the end user is interested in identifying.

[0070]

[94] The engines 132A-N may be arranged in series and / or parallel so that the output data generated by each quality assessment engine 132A-N may be analysed and if the output data of any of the series arranged engines 132A-N fails to meet a predefined quality threshold, the quality assessment system 130 may generate output data indicating that the quality of the indicated food product 108A-N is poor.

[0071]

[95] The quality assessment engine 140 can be configured to assess the overall quality of the food products 108A-N based on the quality metric scores determined by the engines 132A-N. For example, the engine 140 can determine an aggregate quality of each of the food products 108A-N. The engine 140 may determine an aggregate quality of a batch of the food products 108A-N. The engine 140 may determine one or more supply chain changes based on each quality metric score determined by the engines 132A-N. As an illustrative example, the quality assessment engine 132A can measure internal quality, and if the engine 132A generates an output indicating that one or more of the food products 108A-N are spoiled, the quality assessment engine 140 can trigger an action to discard one or more of the food products 108A-N. As another example, if engine 132B measures the shelf life and engine 132B generates an output indicating that one or more of food products 108A-N have a short shelf life, quality assessment engine 140 can trigger updates to the distribution plan of one or more of food products 108A-N so that the food products 108A-N are distributed and sold before their shelf life expires. For example, quality assessment engine 140 can generate instructions to have one or more of food products 108A-N delivered to a vendor that is geographically closest to the facility housing the food products 108A-N to reduce the amount of time the one or more food products 108A-N are in transit. Alternatively or additionally, one or more of food products 108A-N that have a limited shelf life may be directed to a refrigeration unit to extend the life of the food products 108A-N. Many other example actions can be determined and performed based on the quality metric scores determined by engines 132A-N (e.g., the engines can provide a quantitative means of differentiating between different suture removals of food products based on the aggregate quality scores).

[0072]

[96] As described throughout this disclosure, the quality metric scores generated by the engines 132A-N can be numeric, binary, and / or Boolean. The scores can be numeric on a predefined scale. The scores may be values ​​such as "good", "poor", "poor", "acceptable", "excellent", "good", "fair", etc. As another example, the output data can be a vector of one of two different values ​​for each food product 108A-N shown that provides an indication, passing quality, or failing quality of each food product 108A-N. An overall quality score may be determined for each food product 108A-N shown based on the vector of scores output by the engines 132A-N. One or more different values ​​can be defined based on the supply chain and / or the practices of relevant stakeholders in the supply chain.

[0073]

[97] In some implementations, an overall quality score 134A-N may be generated for each of the food products 108A-N in the output image 134. The overall quality scores 134A-N may be viewable by relevant stakeholders in the supply chain. The stakeholders may view the scores 134A-N to determine one or more changes to the supply chain. The quality metric scores generated by the engines 132A-N may be output in one or more other forms for review by the relevant stakeholders. For example, the scores may be output as a vector for each of the food products 108A-N.

[0074]

[98] Figure 2 is a flow chart of an overall process 200 for identifying food quality based on image data (see, e.g., Figure 1A). Process 200 can be performed by computer system 150. Process 200 can also be performed by one or more other computing systems, devices, and / or servers.

[0075]

[99] Referring to process 200, computer system 150 may receive image data at 202. As described throughout, the image data may be received from an imaging device. The image data may also be retrieved from a database, data store, or other repository that stores image data of a particular product. The image data may include images of a particular product whose quality is to be identified based on different features seen from the image data. The image data may also include images of a product whose quality can be identified based on unseen features. Such features, such as dry matter, may be learned from the image data.

[0076]

[0100] The computer system 150 may then identify produce characteristics at 204. As part of the feature identification, the computer system 150 may apply ground truth levels to the image data at 206. Features such as spoilage, mold, ripeness, dryness, etc. may be identified from the image data and labeled accordingly. Some features, such as hardness and dryness, may not be visible from the image data and therefore may be labeled using other techniques, such as destructive techniques involving hardness testers and / or penetrometers.

[0077]

[0101] A model can then be generated for each identified product feature at 208. As described throughout, the model can be generated based on color, shape, texture, spectral response (e.g., in the case of multi-spectral data), anything else that may be visible from the image data and / or anything else that may not be visible from the image data but can be derived as ground truth measurements made on the product.

[0078]

[0102] During run-time, the computer system 150 can then receive image data of a product at 210. The image data can be captured by the imaging device 160. The computer system 150 can apply one or more of the generated models at 212. In some implementations, the computer system 150 can select a model to apply based on the image data received at 210. For example, the computer system 150 can use object recognition techniques to identify the type of product captured by the image data. The computer system 150 can then retrieve one or more models associated with the identified type of product from a data store or other database that stores the generated models. In some implementations, the computer system 150 can determine which model to retrieve using metadata (e.g., country, product type, etc.) that is part of the image data for the product. The models can be applied sequentially or in parallel.

[0079]

[0103] The computer system 150 can determine 214 a quality metric score for the attributes identified by the model. In some implementations, a model can be trained to score the identified attributes. In some implementations, one or more other models or analytical engines can be configured to score the attributes identified by the model applied in 212. In some implementations, an overall / aggregate metric score for the product can also be determined based on all of the attributes identified by the model.

[0080]

[0104] The computer system 150 may output 216 a quality metric score for the produce. As described herein, the score may be output to a user device of a relevant stakeholder in the supply chain of the produce (see, e.g., FIG. 1A). The quality metric score may be represented by a numerical value, a Boolean value, and / or a text. For example, the score may be determined on a scale (e.g., 1 to 100, 1 to 5, etc.). The scale may be determined by the relevant stakeholder in the supply chain. As an example, a quality metric score of 1 may indicate poor quality, and a score of 5 may indicate good or high quality. As another example, the score may be "true" or "false," where "true" may indicate good quality and "false" may indicate poor quality. As yet another example, the score may include "good quality," "poor quality," "average," "ripe," "overripe," or any other descriptive string that may be used to identify the quality of the produce.

[0081]

[0105] The computer system 150 can optionally determine 218 the supply chain changes. In some implementations, a user device and / or relevant stakeholders can determine the supply chain changes, as described with reference to FIG. 1A. In some implementations, the computer system 150 can automatically determine and / or implement the supply chain changes. The changes can be determined based on the quality metric scores of the products.

[0082]

[0106] In some implementations, the computer system 150 may identify an artifact in the image data received at 210. Once the artifact is identified, the computer system 150 may proceed to blocks 212-218 of the process 200. To identify the artifact, the computer system 150 may use a generic object detection model to find all bounding boxes around the artifact in the image data. The generic object detection model may be trained to detect any type of artifact with high accuracy. The computer system 150 may determine whether the artifact is found in the image data. If the artifact is found in the image data, the computer system 150 may return all bounding boxes that have the artifact. The returned bounding boxes may be further processed and analyzed. The returned bounding boxes may then be ordered and processed by the computer system 150 to identify the type of artifact within each bounding box. The computer system 150 may utilize an artifact analyzer and / or one or more machine learning models trained to identify artifacts found within the bounding boxes. A CNN and / or an image classification model may be used to positively identify artifacts within the bounding boxes. Each returned bounding box may be processed in a separate job (e.g., sequentially). In some implementations, erased bounding boxes may be processed in parallel. In some implementations, computer system 150 may identify artifacts in advance by analyzing metadata associated with the image data. Sometimes, for example, the metadata may indicate what type of artifacts are found in the image data. Finally, if no artifacts are found in the image data, the bounding boxes may not be returned for further processing and analysis.

[0083]

[0107] 3 is a flow chart of a process 300 for generating a model that identifies different quality characteristics of a food product based on image data (see, e.g., FIG. 1B). As described throughout this disclosure, multiple models can be generated from a training image dataset. Models can be generated that identify different characteristics associated with different foods, different food types, and / or different subcategories of foods. Process 300 can be performed by computer system 150. Process 300 may also be performed by one or more other computing systems, devices, and / or servers.

[0084]

[0108] Referring to process 300, the computer system 150 can receive image data at 302. The image data can be for a particular product. Image data can also be collected for processed and unprocessed products. The image data can include images of the exterior and / or interior of the product. The image data can also include multiple images of the product throughout the product's lifespan. The image data can include images of the same type of product.

[0085]

[0109] The computer system 150 can identify 304 characteristics of the produce from the image data. For example, object detection techniques can be performed on the image data to detect the produce, and then additional extraction steps can be performed to pick out one or more specific characteristics of the produce. Examples of characteristics can include spoilage, blemishes, dryness, color, mold, texture, etc. Other examples of characteristics include non-visible characteristics including, but not limited to, hardness, dry matter, brix, etc. Each of the specific characteristics extracted from the image data can be labeled.

[0086]

[0110] The computer system 150 may then generate 306 a model of the identified produce characteristics. The model may be generated using machine learning techniques such as CNN. For example, the model may be trained using CNN to identify specific features from the labeled image data. As a result, during run-time, the model may identify specific produce characteristics from image data received during run-time. As another example, partial least squares (PLS) may be used to generate a model that assesses the spectral data. Such a model may be used to identify hardness and / or dry matter.

[0087]

[0111] As an illustrative example, image data of lime can be received (302). Using segmentation and analysis techniques, dryness can be a feature identified by the computer system (304). Image data of lime that shows signs of dryness can be labeled as dryness, while image data of lime that does not show signs of dryness can be labeled as good lime. Using a CNN, a lime dryness model can be trained to differentiate image data that shows dryness from image data that does not show dryness (306). For example, the model can be trained to analyze each patch and / or pixel in the image data to see if lime is present, and if lime is present, based on the labeled image data, whether the lime appears to show signs of dryness. If the lime appears to show signs of dryness, the model can be trained to tag or otherwise classify / label the lime in the image data as dryness. The model may also be trained to tag lime with other descriptors for dryness, including but not limited to numerical, Boolean, and / or string values.

[0088]

[0112] The computer system 150 may then determine (308) whether there are more characteristics of the produce in the image data. For example, object detection techniques may be performed again. Using such techniques, one or more additional characteristics of the produce may be extracted from the image data. In the lime example above, if a first model was generated for a first characteristic, dryness, then a second model may be generated for a second characteristic, color change, and so on. One or more other and / or additional models may be generated for one or more other characteristics discernible from the image data.

[0089]

[0113] If there are additional features, the computer system 150 may return to block 304 and repeat 304-308 for each remaining feature of the produce. Thus, multiple models may be generated based on the collection of image data. Optionally, the computer system 150 may receive additional training image data to generate models that identify additional features of the produce. Generating multiple models may be advantageous to ensure a more robust and accurate quality assessment of the produce. Eventually, each model may be trained to specifically identify and optionally score a particular feature of the produce. Each model may be tuned to identify a particular feature, but may also receive as input the output of the other models. Thus, running the models sequentially may be advantageous to build a more robust and accurate quality assessment of the produce.

[0090]

[0114] If there are no additional features to generate the models, the computer system 150 may output the generated models (310). Outputting the generated models may include presenting the models to a user at a user device. The user may then select one or more of the models to apply during run-time. Outputting the generated models may also include storing the models in a data store or other database. The models may then be retrieved by the computer system 150 during run-time.

[0091]

[0115] 4A and 4B are a flowchart of a process 400 for identifying food quality using a model during run-time. Process 400 can be performed by computer system 150. Process 400 may also be performed by one or more other computing systems, devices, and / or servers.

[0092]

[0116] Referring to process 400 in both FIG. 4A and FIG. 4B, computer system 150 may receive image data of a food item at 402. The image data may represent multiple food items. In some implementations, the image data may represent only one food item. As described throughout, the image data may be received from one or more imaging devices in an environment (e.g., a storage facility) along the supply chain. In some implementations, the image data may be received from a user device, such as a mobile phone, smartphone, laptop, and / or tablet. A sensor (e.g., a camera) of the user device may be configured to capture an image of the food item. For example, a computer may use the user device to capture an image of an apple in a supermarket. This image data may be transmitted from the user device to computer system 150 for processing.

[0093]

[0117] The computer system 150 may then perform object detection techniques to identify a bounding box for each food item in the image data (404). The computer system 150 may also use a CNN or other machine learning trained model to identify a bounding box for each food item. For example, a deep learning object detector may be trained to identify or otherwise locate all foods found in the image data. A bounding box may be generated for each identified food item such that feature and quality analysis may be performed for each identified food item with respect to the bounding box.

[0094]

[0118] The computer system 150 can identify a grid structure based on the bounding boxes at 406. In other words, the computer system 150 can perform indexing and assign an index to each bounding box that makes up the grid structure of the image data. Each identified food item can receive one of the indexes, which can be used to identify the food item. To identify the grid structure, the computer system 150 can (1) find what appears to be most likely to represent a row of food items based on the Y height of the bounding boxes, and then (2) sort the entire data frame based on the X position. Thus, the grid can be identified and each bounding box in the grid structure can be assigned an index value, which can be used to identify the food item seen in the bounding box. One or more machine learning trained models can be used to identify the grid structure and assign the indexes.

[0095]

[0119] Identifying the grid structure at 406 and indexing the structure can be advantageous for associating the identified quality metric scores with the foods found in the image data. Ultimately, each food may have a different quality score, and therefore each determined quality score should be assigned to an index for the corresponding food. Furthermore, assigning index values ​​to foods can be advantageous for correlating additional metrics and data about a particular food with the quality metric scores determined by computer system 150. As a result, computer system 150 can construct a more robust and accurate quality metric score for a particular product.

[0096]

[0120] In some implementations, the grid structure can also be advantageous to facilitate searching through all image data of a food product to identify and / or output food products having particular characteristics and / or quality scores. This can also be beneficial to train the models described herein to more accurately identify and score quality characteristics. This can also be beneficial to relevant stakeholders in the supply chain who are interested in monitoring and / or analyzing the ROI of foods.

[0097]

[0121] The computer system 150 can select 408 a bounding box for a particular food item shown in the image data. In other words, the computer system 150 can use one or more models described herein to select one of the foods shown in the image data for assessment.

[0098]

[0122] The computer system 150 may then sequentially apply one or more machine learning trained models to the selected bounding boxes (410). For example, the computer system 150 may run the bounding box portions of the image data through a CNN model that assesses food quality based on color. As described throughout this disclosure, the computer system 150 may select one or more models to sequentially apply to the bounding boxes and / or food classifications. The models may be selected based on the type of food, user (e.g., stakeholder) preferences, and / or any features that may be desirable in assessing food quality. Each model may be run sequentially independent of each other.

[0099]

[0123] Advantageously, running models sequentially can automate the quality assessment process that may typically require a human operator to visually inspect the quality, size, shape, color, etc. of a particular food product. Running models sequentially can also be advantageous to provide deeper analysis and discovery based on the output of a previously run model. Thus, as an example, the quality metric scores output by a first model can be received as input to a second model. The quality metric scores output by the second model can then be received as input to a third model, and so on.

[0100]

[0124] As another example, the first model can identify the browning of the fruit skin. The first model can then extract features indicative of browning in the image data. These extracted features can be provided as input to a second model, which can be trained to identify the ripeness stage of the fruit based on the browning. The second model's identification of the ripeness stage of the fruit can then be provided as input to a third model. The third model can be trained to determine an overall quality metric score for the food product based on the ripeness stage based on the browning. Thus, a more robust quality metric assessment can be performed. It can be realized that any number of models can be run in any order to perform a robust quality metric assessment.

[0101]

[0125] Further, in some implementations, models can be run and trained to determine whether the image data of the food product should be run through any additional models. For example, a first model can determine whether the food product is spoiled. If the first model determines that the food product is spoiled, the computer system 150 can decide not to run additional models that may analyze the food product for other quality characteristics, such as yellowing, and ultimately, if the food product is deemed to be of poor quality due to the identification of spoilage, applying additional models will not improve the quality score of the food product.

[0102]

[0126] The computer system 150 may determine a quality metric score for the food product at 412. As described throughout, the score may be a numeric value, a Boolean value, and / or a string. The score may be based on a user-defined scale (e.g., 1 to 100), true (e.g., indicating good quality) and false (e.g., indicating poor quality), or other descriptors that may be used to describe the quality of the food product identified from the image data. A quality metric score may also be determined for each model applied to the image data. In some implementations, the computer system 150 may determine an aggregate quality score for the food product based on one or more outputs from the models applied to the image data. Additionally, in some implementations, the quality score for the food product may be output from the last model applied in sequence. For example, if three models are applied in sequence, the output of the first model becomes the input to the second model, the output of the second model becomes the input to the third model, and the output of the third model becomes the quality score for the food product. The computer system 150 may translate the output from the third model into a value representing a quality score. For example, the output of the third model may indicate that spoilage has been detected in the image data. The computer system 150 may translate or otherwise assign the spoilage indication to a number, Boolean value, and / or string representing poor quality.

[0103]

[0127] The computer system 150 can then store (414) the quality metric score of the food using the grid index of the food. The computer system 150 can associate the quality metric score with the grid index assigned to the food in 406. Thus, the quality metric score can be stored in a data store or other type of database. The quality metric score can be retrieved by the computer system 150 and presented to one or more user devices. Thus, the quality metric score can be used for future analysis and monitoring of the particular food. In some implementations, the quality metric score can also be used in future training datasets to refine the model and increase the accuracy of the model.

[0104]

[0128] The computer system 150 may determine whether additional foods are shown in the image (416). For example, the computer system 150 may determine whether there are more indexed bounding boxes in the grid structure that have not yet been assigned a quality metric score. In some implementations, the computer system 150 may check a data store to see which grid indexes have been assigned a quality metric score.

[0105]

[0129] If there are additional foods in the image data, the computer system 150 may return to blocks 408 and repeat 408-416 for each food remaining in the image data. For example, the computer system 150 may select a bounding box having a grid index that has not yet been assigned any quality metric score. Blocks 408-416 may be executed to score the quality of the foods visible in the selected bounding box.

[0106]

[0130] If there are no more additional foods in the image data, the computer system 150 may output (418) the quality metric scores for the foods in the image data. The quality metric scores may be presented in a number of ways. For example, the quality metric scores may be shown using an image or portion of the image data that represents the food. Foods classified as poor quality may be represented by an output that includes a magnified image of a blemish or other feature on the food that may have caused the food to be of poor quality. As another example, the quality metric scores may be shown using a spectrograph, so that the actual color of the food is shown to demonstrate why the food is of poor quality. The quality metric scores may also be output as whatever value the computer system 150 has assigned to it. In other words, the quality metric scores may be output as a number, a Boolean, and / or a string.

[0107]

[0131] In some implementations, the computer system 150 can output one or more quality metric scores for one or more foods based on user preferences. For example, a user at a user device can provide an input to the user device requesting a display of quality metric scores for a subset of foods shown in the image data. The input can also request a display of a subset of the quality metric scores for a particular food represented in the image data. As yet another example, the input can request a display of a quality metric score for a particular food represented in the first image data and a quality metric score for another food represented in the second image data. One or more other user inputs may be used to generate a quality metric output for display on the user device.

[0108]

[0132] Optionally, computer system 150 can determine one or more supply chain changes at 420. As described throughout this disclosure, the changes can include changing the location where the food product is shipped, changing the amount of time the food product is stored, changing the storage conditions, applying a ripening agent or other treatment to the food product, discarding the food product, moving the food product for shipment to an end consumer, moving the food product for shipment to a food processing plant, etc. For example, if the food product is scored as poor quality, computer system 150 can determine that the food product can be discarded, shipped to a food processing plant, and / or delivered to a supermarket that is geographically closest to the food product's current location. As another example, if the food product is scored as good quality, computer system 150 can determine that the food product can be stored for a longer period of time than other foods and / or that the food product can be shipped to a supermarket that is geographically distant from the food product's current location. As yet another example, if the food product is scored as beginning a ripening process, computer system 150 can determine that the food product can begin transportation to a destination location, that a ripening agent or other treatment can be applied to the food product, that the food storage conditions can be changed, and / or that the food product can be stored for a longer period of time. One or more other modifications may also be possible based on what quality score has been assigned to the food product.One or more other modifications may also be possible based on user-specified preferences associated with different quality metrics.

[0109]

[0133] 5 is another flow chart of a process 500 for assessing the quality of one or more food products (see, e.g., FIG. 1C). Process 500 can be performed by computer system 150. Process 500 may also be performed by one or more other computing systems, devices, and / or servers.

[0110]

[0134] Referring to process 500, computer system 150 can receive image data of the quality of food in an environment at 502. For example, computer system 150 can obtain first data representing an image. The image can include (i) a first portion depicting one or more food items and (ii) a second portion depicting at least a portion of an environment in which the one or more food items reside.

[0111]

[0135] The computer system 150 can index foods in the image data, as described herein, at 504. The computer system 150 can identify portions of the image having foods, at 506. For example, this can include identifying portions of the first data that correspond to areas of the image that are represented by the first portion showing one or more foods.

[0112]

[0136] Next, computer system 150 may identify sub-portions of the portion, each of which includes a food item (508). For example, computer system 150 may identify sub-portions of the identified portion of the first data. Each sub-portion may correspond to an area of ​​the portion of the first data in which a particular food item of the one or more food items is depicted.

[0113]

[0137] The computer system 150 can input each sub-portion to a quality assessment system at 510. In other words, the computer system 150 can sequentially apply the machine learning trained model to each sub-portion (512). The quality assessment system can include one or more models trained to identify and / or score different quality metrics of foods depicted in the first data, as described throughout this disclosure. Thus, the computer system 150 can provide the identified portions of the first data as inputs to the food quality assessment system. The food quality assessment system can include one or more quality assessment engines that can be configured to generate output data indicative of quality attributes of the food depicted by the captured images. The quality assessment system can be part of the computer system 150. In some implementations, the quality assessment system can be different / separate from the computer system 150.

[0114]

[0138] The computer system 150 may then determine a quality metric score for each food item in each subportion (514). For example, the computer system 150 may determine a quality assessment score for each particular food item of the one or more food items based on output data generated by one or more quality assessment engines of the food quality assessment system. Thus, the computer system 150 may determine a quality score for each food item. In some implementations, the computer system 150 may determine an aggregate quality score for the batch of food items represented in the captured images.

[0115]

[0139] The computer system 150 can store (516) the quality metric scores determined for each food item based on the food item's index. As described throughout, the scores can be stored in a data store, database, cloud storage, or other type of memory device. The scores can later be retrieved by the computer system 150 or one or more other computing systems / devices for further analysis, monitoring, and output operations.

[0116]

[0140] 6 is an example of a system diagram of components used to identify food quality based on the techniques described herein. The computer system 150, the food quality data store 600, and the model data store 660 can be in communication (e.g., wired and / or wireless) over a network 180.

[0117]

[0141] Computer system 150 may include an object detection engine 602 (e.g., the food identification engine 120 of FIG. 1C, the extraction engine of FIG. 1C, and / or a combination of the food identification engine 120 and the extraction engine 110), an indexing engine 604, a quality assessment engine 606 (e.g., the quality assessment system 130 of FIG. 1C), a quality assessment engine 608 (e.g., the quality assessment engine 140 of FIG. 1C), and a model training engine 609. In some implementations, one or more of the components 602, 604, 606, 608, and 609 may be separate from computer system 150 and may be part of one or more other computer systems, computers, servers, devices, and / or networks.

[0118]

[0142] The model training engine 609 can be configured to generate one or more models that can be used by the quality assessment engine 606 (see, e.g., FIG. 3). The engine 609 can receive training image data of food products. The engine 609 can identify features associated with the food products from the training image data. The identified features can be annotated or otherwise labeled. A model can then be trained using machine learning techniques (e.g., CNN) to identify the labeled features in the image data. As described throughout this disclosure, the engine 609 can also train one or more models to score the identified features.

[0119]

[0143] The generated and trained models can be stored in the model data store 660 as quality models 662A-N. The quality models 662A-N can be accessed and / or retrieved by one or more analyzers of the quality assessment engine 606 during run-time. As described throughout this disclosure, only some of the models 662A-N can be selected during run-time based on a user's preferences for the types of foods in the image data and / or the quality assessment of specific foods in the image data. In some implementations, the quality models 662A-N can be updated or otherwise changed based on run-time application of the models to the image data.

[0120]

[0144] The object detection engine 602 can be configured to detect one or more foods in the image data as described throughout this disclosure (e.g., see FIGS. 4A and 4B). The engine 602 can receive the image data and perform object detection techniques to process the image data. The engine 602 can also apply one or more machine learning models that are trained to identify the foods in the image data. The engine 602 can generate a bounding box around each food in the image data. For each food 650A-N, the engine 602 can store a bounding box image of the food 656 in the food quality data store 600.

[0121]

[0145] In some implementations, the object detection engine 602 may calibrate the color of the bounding box portions of the image data to maintain color consistency throughout the image data. In some implementations, color calibration may be performed by a separate engine, such as a color calibration engine (not shown). Color calibration may be applied to the entire image data as a pre-processing step before object detection and / or object extraction. Once the image data has been calibrated based on color, RGB image analysis may be performed as described throughout this disclosure.

[0122]

[0146] The indexing engine 604 can be configured to fit a grid structure to the image data and index each bounding box within the structure (see, for example, Figures 4A and 4B). Each bounding box can encompass a food item represented in the image data. For each food item 650A-N, the engine 604 can retrieve a bounding box image 656 from the food quality data store 600 and assign an index value to the bounding box. Thus, each food item can be identified by the assigned index and can be useful for future retrieval, analysis, and processing operations of information. For each food item 650A-N, the engine 604 can store a grid index 658 in the food quality data store 600.

[0123]

[0147] The quality assessment engine 606 may be configured to identify one or more features indicative of the quality of the food products depicted in the image data (see, for example, FIGS. 4A and 4B). The engine 606 may retrieve a bounding box image 656 of each food product 650A-N from the food product quality data store 600. The bounding box image 656 may be used by the analyzer in identifying the quality of the food products. The engine 606 may also retrieve one or more quality models 662A-N from the model data store 660 for execution at run-time. The engine 606 may select the quality model 662A-N based on a determination made by the object detection engine 602. For example, the engine 602 may determine that the food products identified in the image data are all apples. The engine 602 may inform the quality assessment engine 606 that the food products are apples. In some implementations, the food product identifications may be provided to the quality assessment engine 606 from a computing system, computer, device, network of data stores and / or cloud-based services. The quality assessment engine 606 may then retrieve one or more quality models 662A-N associated with the apples from the model data store 660. Once retrieved, the models 662A-N may be executed in sequence by one or more of the analyzers.

[0124]

[0148] Although the analyzers are shown and described in one order in FIG. 6, the analyzers may be shown and executed in any one or more other orders. The order of execution may be determined by the quality assessment engine 606. The order of execution may be determined by a user at a user device. In some implementations, the order of execution may be determined by the computer system 150. The order of execution may be based on the type of food in the image data. The order of execution may be based on the particular quality characteristic being assessed. Additionally, the order of execution may be based on user preference.

[0125]

[0149] The quality assessment engine 606 can include multiple analyzers, each of which can be configured to identify a different quality characteristic in a food product based on the image data. Thus, each model 662A-N can be performed by a different analyzer. Although some examples are shown in FIG. 6, additional analyzers may be included in the engine 606. Thus, the engine 606 may include one or more additional or fewer analyzers. Each analyzer can receive, for each food product 650A-N, a bounding box image 656 for that food product.

[0126]

[0150] By way of example, the quality assessment engine 606 may include a color analyzer 610, an apple yellowing analyzer 612, a lime yellowing analyzer 614, an avocado internal analyzer 616, a lime rot and drying analyzer 618, a shelf life analyzer 620, a produce grade analyzer 622, a size analyzer 624, a shape analyzer 626, a ripening coating analyzer 628, a dry matter analyzer 630, a hardness analyzer 632, a Brix analyzer 634, a nutritional analyzer 636, a sticker analyzer 638, a sourness analyzer 640, a general internal defect analyzer 642, a general color analyzer 644, a wrinkle analyzer 670, a calyx browning analyzer 672, a banana stage analyzer 674, a cherry stem color analyzer 676, and / or a general external defect analyzer 678. The one or more additional analyzers may include a lenticel oxidation (black spot) analyzer (e.g., for mangoes) or other types of analyzers that may be generally applicable to identifying quality attributes for different types of foods and / or that may be specifically applicable to certain types of foods. More specifically, analyzers 612, 614, 644, 672, 674, and 676 may be a subset of color analyzer 610 and are merely illustrative examples of color analyzers. Additionally, any of analyzers 672, 674, and / or 676 may be used to identify and extract a median color value that may then be used to identify quality attributes of a food entity (e.g., a cherry entity or surface using analyzer 676, a strawberry entity or surface using analyzer 672). The techniques described with reference to analyzers 610, 612, 614, 644, 672, 674, and 676 may be applied to other types of color analyzers used to identify the quality of other types of foods based on the type of food and / or the color of the food. Similarly, avocado internal analyzer 616 may also be a subset of general internal defect analyzer 642 and may simply be an illustrative example of an internal defect analyzer. One or more other internal defect analyzers may be generated that assess internal defects based on the type of food and / or the type of internal defect. Additionally, lime rot and dryness analyzer 618, dry matter analyzer 630, hardness analyzer 632, ripening coating analyzer 628, and wrinkle analyzer 670 may be subset analyzers of general external defect analyzer 678 and may simply be illustrative examples of external defect analyzers.One or more other external defect analyzers may be created that assess external defects based on the type of food product and / or the type of external defect.

[0127]

[0151] The output from any one or more of the analyzers can be stored in the food quality data store 600 as an analyzer quality score 652A-N for each food product 650A-N. In some implementations, the output from any one or more of the analyzers may be received as input to one or more of the analyzers as described throughout this disclosure.

[0128]

[0152] With reference to the exemplary analyzer shown in FIG. 6, the color analyzer 610 can be configured to analyze the bounding box image 656 to identify the color of the food and the quality feature to which the color corresponds. The color analyzer 610 can be applied to any type of food in the image data. The color analyzer 610 may be configured to identify which colors correspond to different quality features of different types of food. For example, the analyzer 610 can determine that yellowing of an apple indicates poor quality of the apple, while yellowing of a lemon can indicate good quality for a lime. Using a machine learning trained model, the color analyzer 610 can identify a median Euclidean distance in the color space represented by the bounding box image 656 from a reference color. The reference color can be different for each type of food that can be analyzed by the color analyzer. In some implementations, the analyzer 610 can process the image 656 of the food and output data indicative of the median distance of the food from the reference color. This output can then be received by one or more other analyzers. The output data may include an indication of the quality of the food, such as good quality or poor quality.

[0129]

[0153] As an illustrative example, in some implementations, the analyzer 610 can process all image data for a particular food by identifying color samples from each food (e.g., limes, apples, avocados, and other types of produce where color can be a feature that can be used to identify produce quality). All possible colors that can occur in the food can then be mapped into a color space (e.g., CIELAB space or other three-dimensional color space). The analyzer 610 can then reduce this multi-dimensional space to a one-dimensional space by identifying the direction in which one or more of the color samples change the most (e.g., exceeding a color change threshold level) and predicting (e.g., forecasting) the color change along the identified direction for each food. As a result, the analyzer 610 can identify a one-dimensional quality metric for each food based on the predicted color change of the food. The one-dimensional quality metric can be a number ranging from 4 to 80. One or more other ranges may be used, which can be based on the type of produce being imaged. The analyzer 610 can then identify a quality level for each food as good quality based on determining that the food's one-dimensional quality metric is within the threshold quality range. The analyzer 610 may also identify a quality level as poor quality based on determining that the one-dimensional quality metric of the food product is not within a threshold quality range. The threshold quality range may be based on historical data, such as customer preferences, the quality and / or color of foods that customers typically purchase and / or consume, and / or other factors. This process may be effective in accurately assessing and scoring the color of different types of foods. This process may also be computationally easy and fast, thereby reducing the time and computational resources utilized to identify food quality based on color.

[0130]

[0154] The apple yellowing analyzer 612 may be configured to analyze food products, such as Granny Smith apples, to identify the degree of yellowing of such food products. In some implementations, a Gaussian mixture model may be used to assess the yellowing of apples. Thus, the analyzer 612 may be configured to detect food products of poor quality. The analyzer 612 may process the output from the bounding box image 656 or one or more other analyzers to generate output data indicative of a yellowing score of the food product. In some implementations, the output may be a numerical score in the range of 0-100, with 0 indicating that the apple is not yellow and 100 indicating that it is the maximum shade of yellow (e.g., the yellowest). An apple receiving a score of 100 may be too ripe and therefore of poor quality, while an apple receiving a score of 0 may not yet be ripe and may be of good quality (or the quality is not yet known because it is too early in the apple's life). In some implementations, the apple yellowing analyzer 612 may perform similar processing techniques as the analyzer 610 described above.

[0131]

[0155] As an illustrative example, a color analyzer 610 may be run on an image of an apple. The color analyzer 610 may generate an output indicating that the apple is predominantly yellow in color. However, the analyzer 610 may not be trained to glean additional insight into what it means to be predominantly yellow. Thus, an output from the color analyzer 610, which may be a patch of the image of the apple that contains predominantly yellow, may be received as an input to an apple yellowing analyzer 612. The analyzer 612 may be trained to assess the predominant yellow color and determine what that color means in terms of the quality of the apple. Thus, the analyzer 612 may score the predominant yellow color and output the score. The analyzer 612 may also output an indication that the apple is of good or poor quality based on the score.

[0132]

[0156] In another scenario using the above example, if the color analyzer 610 produces an output indicating that the color of the apple is green throughout the image, the color analyzer 610 can be trained to produce an output indicating that the apple is of good quality. As a result, the apple yellowing analyzer 612 may not need to be run because the analyzer 612 returns a score of 0 that simply confirms the assessment of the analyzer 610. Since the analyzer 612 does not have to be run, processing time can be reduced. Computational resources can also be allocated to running other models and assessing features in other foods. In some implementations, the analyzer 612 may still be run sequentially as a means of confirming the specific accuracy of the color analyzer 610.

[0133]

[0157] Still referring to the analyzers of the quality assessment engine 606, the lime yellowing analyzer 614 can be configured to analyze food products, such as limes, from image data to identify the degree of yellowing of such limes. The engine 614 can generate output data indicative of a yellowing score for each lime. For example, the output score can be within a range of values ​​(e.g., 0 to 4, with 0 indicating that the lime is not yellow and 4 indicating that the lime is at its maximum shade of yellow). One or more different ranges of values ​​can be used. The engine 614 can also determine that limes that are at their maximum shade of yellow are overripe and therefore of poor quality. The engine 614 can be trained to grade the degree of yellowing in limes using a Gaussian mixture model and a support vector regressor.

[0134]

[0158] The avocado internal analyzer 616 can be configured to analyze image data of cut-open avocados to identify an overall quality of the food product. For example, some image data can include images of cut-open food products. These images can be analyzed by the analyzer 616. In some implementations, the analyzer 616 can generate output data indicative of whether each cut-open avocado is of excellent, medium, or poor quality. The overall quality score of excellent, medium, or poor quality can be expressed as a score (e.g., a numerical and / or Boolean value) that can be mapped to one or more of the above quality categories of excellent, medium, or poor. Such a score can be determined based on several intermediate values ​​generated by the analyzer 616. In such implementations, the intermediate values ​​can include Boolean values ​​specific to features such as extensive browning, vascular browning, pit delamination, internal rot, stem dieback, tissue destruction, or any combination thereof. The intermediate values ​​can include numerical values ​​such as probability values ​​within one or more ranges (e.g., a range of 0 to 1). Additionally, the analyzer 616 can run one or more different machine learning trained models, each configured to identify different features listed above. In some implementations, the analyzer 616 can run one model that can be trained to identify any combination of the features listed above.

[0135]

[0159] The lime spoilage and drying analyzer 618 can be configured to analyze food products such as limes to identify a level of lime spoilage and / or drying shown in the image data. The analyzer 618 can generate output data indicative of a binary or Boolean classification of spoilage, drying, or both. The analyzer 618 can be trained to identify features in the image 656 that can be early indicators of spoilage and / or drying and current spoilage and / or drying.

[0136]

[0160] The shelf life analyzer 620 can be configured to analyze food products to infer or predict the level of ripeness (e.g., maturity), remaining shelf life, or both under one or more predefined conditions. The analyzer 620 can generate output data indicative of a score based on a ripeness scale. In some implementations, the score can be mapped to a particular number of days remaining until each food product is ready to eat. The ripeness level can be determined based on color and / or food quality. Ripeness can also be determined from hyperspectral and / or multispectral images by identifying the hardness and / or dry matter of the food product. As an illustrative example, ripeness correlates with hardness (e.g., the softer the avocado, the riper it is). Thus, a hardness analyzer can be used to assess the hyperspectral image data and determine a hardness value. The hardness value can then be correlated to the ripeness level of the avocado.

[0137]

[0161] Produce grade analyzer 622 may be configured to analyze the food product represented in bounding box image 656 to determine a grade, such as a USDA specification grade. Analyzer 622 may generate output data indicative of a score for the food product that correlates the quality of the food product with the USDA designated grade. The score may indicate an assessment of the food product's color, shape, fineness, and / or any combination thereof.

[0138]

[0162] The size analyzer 624 can be configured to identify a size of the food from the bounding box image 656. The analyzer 624 can generate output data indicative of a dimension of the food. The size analyzer 624 can be trained to identify reference points in the image data and identify a size of the food within the bounding box relative to the reference points. In some implementations, the output data indicative of the size of the food can be evaluated to determine whether the food is within a predetermined size range of a standard food binning. The data indicative of the size of the food can be represented using a Boolean value. For example, if the food is within the standard binning size range, a Boolean value such as 1 can be generated for the food. Alternatively, in such implementations, if the food size is not within the standard binning size range, a value of 0 can be generated for the food. In some implementations, the output data can be the identified dimension of the food. Further, in some implementations, the data indicative of the size of the food can be represented using a numeric value. For example, avocados can be represented by a variety of number size values, including, but not limited to, 28 (13.75-15.70 oz), 32 (11.75-14 oz), 36 (10.50-12.50 oz), 40 (9.50-11.50 oz), 48 (7.50-9.50 oz), 60 (6.25-7.50 oz), 70 (4.75-6.25 oz), and 84 (3.75-4.75 oz).

[0139]

[0163] The shape analyzer 624 can be configured to determine whether the shape of the food product depicted in the image 656 matches a predetermined shape metric for that particular type of food product. The analyzer 624 can be trained to identify the shape of the food product in the image 656 and compare the shape to a desired shape for the same type of food product. For example, image data of a cucumber can be analyzed to determine whether the cucumber has a particular level of curvature. A cucumber that has the desired level of curvature can be identified as good quality, while a cucumber that does not have the desired level of curvature can be identified as poor quality. One or more other identifications may be made based on whether the food product meets the desired shape metric. The analyzer 624 can generate an output, such as a score for the food product. The output data can also indicate the degree to which the predetermined shape metric is embodied in the food product and / or the degree to which the shape of the food product deviates from the desired shape.

[0140]

[0164] General external defect analyzer 678 can be configured to identify the presence of one or more external defects on the surface of the food product shown in image 656. Analyzer 656 can implement one or more machine learning models configured to identify and assess visible and / or infrared light in image 656 to determine whether one or more external defects are present on the surface of the food product. For further discussion of general external defect analyzer 678, see the discussion of analyzers 618, 628, 630, 632, and 670.

[0141]

[0165] The aged coating analyzer 628 can be configured to identify the presence, thickness, absence, or combinations thereof, of an aged coating on the food product shown in the image 656. The spectral reflectance profile can be used by the analyzer 628 to identify a coating on the surface of the food product. The spectral reflectance profile can be measured using a multispectral and / or hyperspectral imaging device. The analyzer 628 can generate output data indicative of whether the food product has an aged coating, the thickness of the aged coating, or the absence of an aged coating. In some implementations, output data such as a numerical value of 0 can indicate the absence of a coating, while any value greater than 0 and less than 1 can indicate the presence of a coating and a corresponding value of the coating. For example, an output value closer to 1 can indicate a thicker aged coating than a value closer to 0.

[0142]

[0166] The dry matter analyzer 630 can be configured to identify the dry matter level in the food product shown in the image 656. The hyperspectral image can be used by the analyzer 630 to identify the dry matter level. For example, the analyzer 630 can implement a machine learning trained model that uses visible and infrared spectral reflectance data to identify and quantify dry matter. The analyzer 630 can generate output data, such as a score, indicative of the level of dry matter. If the output data is numeric, a value of 0 can indicate no dry matter in the food product, while any value greater than 0 and less than 1 can indicate the level of dry matter found in the food product. Thus, an output value closer to 1 can indicate the presence or amount of more dry matter in the food product than a value closer to 0.

[0143]

[0167] The hardness analyzer 632 can be configured to identify a hardness level of the food depicted in the image 656. The hyperspectral image data can be used by the analyzer 632 to identify the hardness level. For example, the analyzer 632 can implement a machine learning trained model that uses visible and infrared spectral reflectance data to identify foods and quantify the hardness of the food. The analyzer 632 can generate output data, such as a score, indicative of the hardness level. The output data can be a number in a range, such as 0 to 1. A value of 0 can indicate that the depicted food is hard and inedible, while any value greater than 0 and less than 1 can provide an indication of hardness that is not hard. For example, an output value closer to 1 can indicate that the food is soft, not hard, and / or that the food is approaching the maximum softness that it should be. In some implementations, depending on the type of food, the maximum softness can indicate that the food is of good quality and ready to be consumed by the consumer. In some implementations, the maximum softness can indicate that the food is of poor quality and is no longer of the desired hardness to be consumed by the consumer.

[0144]

[0168] The Brix analyzer 634 can be configured to identify the sugar content of the food in the image 656. The hyperspectral image data can be used by the analyzer 634 to identify the sugar content. For example, the analyzer 634 can implement a machine learning trained model that uses the visible and infrared spectral reflectance data to identify the food and quantify the sugar content associated with the food. The analyzer 634 can generate output data, such as a score indicative of the sugar content. The score can be a number on a scale. For example, the scale can be from 0 to 1, where 0 can indicate no sugar is detected in the food. Any value greater than 0 and less than 1 can indicate the sugar content of the food, where a value closer to 1 can indicate a greater sugar content in the food. The analyzer 634 can also be configured to identify and output a sweetness value or a sourness value of the food in the image 656. For example, a value closer to 1 can indicate the food is sweeter than a food having a value closer to 0. In some implementations, 1 can indicate a maximum sweetness desired for the food. In some implementations, 1 can indicate a maximum sourness desired for the food. The analyzer 634 may also identify the quality of the food product based on the maximum sweetness or sourness of the food product. For example, if the food product has a value of 1, indicating maximum sweetness or sourness, the analyzer 634 may determine that the food product is of poor quality because, depending on the type of food product, a customer may not want to eat something that is too sweet or too sour. As another illustrative example, if the food product has a value of 0.5, the analyzer 634 may determine that the food product is of a preferred sweetness or sourness and therefore of good quality.

[0145]

[0169] The nutritional analyzer 636 can be configured to identify the nutritional level of the food. The hyperspectral image data can be used by the analyzer 636 to identify the nutritional content. For example, the analyzer 636 can implement a machine learning trained model that uses visible and infrared spectral reflectance data to identify the food and quantify the nutritional content of the food. The analyzer 636 can generate the output data as a number on a scale. For example, on a scale of 0 to 1, 0 can indicate no nutritional value of the food, and any value greater than 0 and less than 1 can indicate a nutritional level of the food.

[0146]

[0170] The sticker analyzer 638 can be configured to detect and interpret information on a sticker or other label attached to the food product. The image 656 can detect foods that have a label attached to their surface. If a label is present, the analyzer 638 can be trained to detect the label. Using image processing and optical character recognition (OCR) techniques, the analyzer 638 can read the label and glean additional information about the food product. For example, the analyzer 638 can identify information such as the name of the food product, a barcode or other product identifier associated with the food product, a customer name or identifier, a sale date, a best before date, a place of origin, a destination location, and the like. The analyzer 638 can generate output data that includes the decoded text or identifier that was on the label. The output can also include a magnified image of the label extracted from the image 656 by the analyzer 638.

[0147]

[0171] The acidity analyzer 640 can be configured to identify the acidity level of the food shown in the image 656. The hyperspectral image can be used by the analyzer 640 to identify the acidity level (pH). For example, the analyzer 640 can implement a machine learning trained model that uses visible and infrared spectral reflectance data to identify foods and quantify the acidity level of the food. The analyzer 640 can generate output data as a score indicative of the acidity level, which can be a number on a scale. An example scale can be from 0 to 1, where a value of 0 can indicate no acid in the food. A value greater than 0 and less than 1 can indicate a level of acid in the food.

[0148]

[0172] The general internal defect analyzer 642 may be configured to analyze the intact food product to identify the overall quality of the internal portion of the food product in the image 656. For example, the analyzer 642 may receive a hyperspectral image or other hyperspectral image data of the food product. Using the hyperspectral data, the analyzer 642 may generate output data indicating whether the intact food product has internal defects. One or more machine learning models may be used to assess the hyperspectral data and identify internal defects. For example, hyperspectral imaging utilizes different portions of the electromagnetic spectrum. Some bands within that spectrum are capable of penetrating into certain materials. The penetration depth depends on the band and the material being imaged. Some foods, for example, spectral bands may penetrate and interact with the interior of a fruit such that the image of the food product that is generated contains information about the internal condition of the fruit. This allows the analyzer 642 to detect internal defects using one or more machine learning trained models. In some implementations, the analyzer 642 may also identify those spectral features in the flesh of the food product that are present with and / or without internal defects. Internal defects may include blemishes, spoilage, disease, and the like. The analyzer 642 can be trained to identify internal defects in any type of food product. The output data generated can include a Boolean or binary classification indicating whether the food product has an internal defect or not. The output data can also indicate what type of internal defect may be present in the food product.

[0149]

[0173] The general color analyzer 644 can be configured to identify a grade level and other quality characteristics of a food product in the image 656 based on an analysis of the color of the food product. The analyzer 644 can be trained to identify grade levels and quality characteristics of any type of food product. The analyzer 644 can generate output data indicative of the grade of the food product, which can be a numerical value. The numerical value can be mapped to one or more overall food quality categories. Such categories can include excellent, fair, and poor quality. The numerical value may also be mapped to one or more other quality descriptors.

[0150]

[0174] The analyzer 644 can be trained using images of different types of food labeled with specific food quality categories. Additionally, the analyzer 644 can be trained to identify the quality of food based on an RGB model of the food in the image 656. The analyzer 644 can generate output data as a Boolean flag indicating whether the food meets the RGB quality test. The analyzer 644 can be trained to identify color features in the image 656 on a pixel-by-pixel basis. As another example, the analyzer 644 can be trained to identify the quality of food based on a hyperspectral image (HSI) of the food. The analyzer 644 can generate output data as a Boolean flag indicating whether the food meets the HSI quality test. The analyzer 644 can be trained to perform hyperspectral cube processing to identify the quality of food from the image 656.

[0151]

[0175] The wrinkle analyzer 670 can be trained to identify the percentage of wrinkle coverage on a food product. The food product can include mango. Additionally or alternatively, the food product can include bell pepper. The wrinkle analyzer 670 may be applied to analyze one or more other types of food products whose quality may be defined or otherwise affected based on the amount of wrinkles on the surface of the food product. The analyzer 670 can employ machine learning algorithms, models, or other machine learning techniques to identify and quantify the amount of wrinkles found on the surface of the food product. Using these techniques, the analyzer 670 can be trained to predict the total percentage of wrinkle coverage on the food product from the image 656. In some implementations, the analyzer 670 can perform a binary classification using machine learning techniques to determine whether the food product is wrinkled (e.g., assign a binary value of 1) or not (e.g., assign a binary value of 0). As an illustrative example, the wrinkle analyzer 670 can use image processing techniques to slice the image 656 into patches, boxes, or other predefined zones. The analyzer 670 may apply one or more machine learning techniques, such as a trained classifier, to each patch to determine whether the patch has wrinkles. Once each patch is analyzed, the analyzer 670 may generate an overall wrinkle coverage score for the food product by averaging, adding up, or otherwise combining all the patches. In some implementations, the overall wrinkle coverage score may be a percentage of overall wrinkle coverage. See FIG. 19 for further discussion.

[0152]

[0176] The calyx browning analyzer 672 can be trained to identify calyx browning in foods such as strawberries. Over time, strawberries can develop calyx browning, which can indicate a low quality level of the strawberries. To identify color features in the strawberry image data that are indicative of calyx browning, one or more of the image processing techniques described herein can be used by the analyzer 672. For example, the strawberry image data can be color calibrated and object detection techniques described herein can be applied to generate bounding boxes around each strawberry found in the image data. Color analysis techniques described herein can be applied to each bounding box to classify the color of the strawberry within the bounding box. Additionally, the analyzer 672 can implement algorithms to separate various aspects or parts of a strawberry, including but not limited to the calyx, flesh, seeds, etc., and then separate and separately quantify the overall color of each of those aspects / parts. Thus, the analyzer 672 can be used to separate and grade the quality of specific aspects or parts of cherries, such as the cherry stem analyzer 676, described further below. A statistical distribution (e.g., average, find median color value, find mean color value, eliminate outlier color values) may be performed on the classified colors to identify a statistical mean color value of the strawberry. The analyzer 672 may then perform a binary decision of whether the strawberry has calyx browning or not based on the statistical mean color value (e.g., if the statistical mean color value is within a threshold color value or a threshold color value range, the strawberry is classified as having calyx browning). The analyzer 672 may assign a quality score to the strawberry represented by the image data based on the binary decision. The quality score may be a string value such as "good," "acceptable," and "poor." As an illustrative example, if the analyzer 672 determines that the statistical mean color value does not match the threshold color value, the strawberry may be assigned a quality score of "good." As another example, if the analyzer 672 determines that the statistical mean color value is within a threshold color value range, the strawberry may be assigned a quality score of "acceptable." As another example, if the analyzer 672 determines that the statistical mean color value matches the threshold color value for calyx browning, a quality score of "poor" may be assigned.

[0153]

[0177] The analyzer 672 can implement one or more machine learning trained rules, techniques, and / or models to analyze strawberries in the image data as described herein. For example, the analyzer 672 can identify a median color value of the food item based on applying the calyx browning trained model to a bounding box that includes the food item and determine a quality level score of the food item based on the median color value of the food item. The quality level score can be assigned (i) a string value of "good" based on the median color value being less than a first threshold color range, (ii) a string value of "acceptable" based on the median color value being greater than the first threshold color range and less than a second threshold color range, and (iii) a string value of "poor" based on the median color value being greater than the second threshold color range. See FIG. 20 for further discussion.

[0154]

[0178] The banana stage analyzer 674 can be trained to identify the r stages of bananas from the image data to identify the quality level of the banana. Consumers may determine the ripeness of bananas, and therefore their purchase decision, based on the color, brown spots, and / or hardness of the banana skin. Thus, the appearance of the banana skin can have a high importance in the consumer decision to purchase a banana or not. The color of bananas changes over time from all green to bright yellow with spots, thereby indicating different levels of ripeness or freshness. Thus, any of the color analysis techniques described herein can be applied by the analyzer 674 to identify the color of the banana skin and correlate the ripeness stage to the identified color. The analyzer 674 can implement machine learning trained models to perform these techniques. The model can be trained, for example, with training data showing bananas annotated at various different ripeness stages. Thus, the model can identify the ripeness stage of bananas from the color analysis of the banana image data. As an illustrative example, the analyzer 674 may identify color values ​​of the food based on applying the banana stage analyzer trained model to a bounding box containing the food in the image data, and assign a quality level score to the food based on mapping the color values ​​of the food to threshold color values ​​expected at multiple ripeness stages. The quality level score may be a string value indicating the current ripeness stage of the food. See FIG. 21 for further discussion.

[0155]

[0179] The cherry stem color analyzer 676 can be trained to identify cherry stems in the image data, classify their color, and identify the quality level of the cherries. The analyzer 676 can employ one or more of the image processing techniques, color analysis techniques, and / or machine learning techniques described herein to analyze the cherry stems. For example, the analyzer 676 can apply a machine learning trained model to the cherry image data to perform the disclosed techniques. Using object detection techniques and / or models, the analyzer 676 can identify the cherries in the image data and generate a bounding box around the cherries. The analyzer 676 can then use object detection techniques to isolate the stems in the bounding box and mask the stems in the original image data for color quantification purposes. The analyzer 676 can identify the average or mean calculated color value of the stems and then correlate the average color value with a quality score. For example, the quality score can be a string value of "good," "poor," or "acceptable." An average color value that meets the first threshold color value range may be assigned a quality score of "good." An average color value that meets the second threshold color value range but not the first threshold color value range may be assigned a quality score of "acceptable." An average color value that meets the third threshold color value (and optionally also the second threshold color value range) but not the first threshold color value range may be assigned a quality score of "poor." One or more other techniques described herein may be used to quantify the color values ​​of the cherry stems and identify the quality level of the cherries. See Figures 22A and 22B for further discussion.

[0156]

[0180] As described throughout, the example analyzers above do not serve to limit the scope of the present disclosure. Instead, any type of analyzer may be used by the quality assessment engine 606. Similarly, the input and output characteristics of each of the example analyzers should not be viewed as limiting. The analyzers may be configured and / or trained to generate other types of output data and receive other types of input data. As an illustrative example, one or more of the above analyzers may be trained to generate different ranges of numerical scores. The different ranges of values ​​may also be assigned different quality indicators based on the food, the type of food, the quality characteristic being identified, and / or user preferences.

[0157]

[0181] Still referring to computer system 150 of FIG. 6, quality evaluation engine 608 can be configured to receive output from one or more of the analyzers of quality assessment engine 606. Engine 608 can translate the output into a quality metric score. As described herein, the score can be a numeric, Boolean, and / or string value. Engine 608 can also be configured to generate supply chain changes based on the output from the one or more analyzers. Additionally, engine 608 can be configured to generate an overall quality score for each food item represented in the image data. Engine 608 can also determine which information / scores are output for display on the user device. For each food item 650A-N, engine 608 can store analyzer quality scores 652A-N (e.g., outputs from the analyzers and / or scores generated by engine 608 based on the analyzer outputs) and an overall quality score 654 in food quality data store 600.

[0158]

[0182] As an illustrative example, the quality assessment engine 608 can determine an overall quality metric for a food product based on applying nested rules across the output from one or more analyzers described above to determine a quality category (e.g., bucket) to assign the overall quality metric to. See FIG. 23 for further discussion. In some implementations, the engine 608 can determine the overall quality metric based on weighting and blending decisions or outputs from the analyzers. In other words, based on the type of food product, the quality assessment engine 608 can combine the received decisions or outputs to generate an overall quality metric or score for the particular food product (e.g., an overall quality score 654 for each food product 650A-N).

[0159]

[0183] As an illustrative example, for mango, the quality assessment engine 608 may receive a color score from the color analyzer 610 and a wrinkle coverage score from the wrinkle analyzer 670. A consumer may be more concerned about wrinkle defects in mango when making a purchasing decision. Thus, since the quality, edibility, and / or saleability of a mango depends more on how wrinkled the surface of the mango is than on the color of the mango, the quality assessment engine 608 may apply one or more rules specific to the mango that weight the wrinkle coverage score more heavily than the color score to determine an overall quality metric for the mango. Based on this weighting, a binary decision may be made by the quality assessment engine 608 as to whether the analyzed mango is saleable / unsellable and / or edible / inedible. The binary decision may be made based on applying one or more rules to each score and / or combination of scores provided for the food product, such as applying a saleability threshold that may depend on the type of food product. Salability thresholds, in some implementations, also vary based on how consumers rank various quality attributes (eg, mold, wrinkles, color, sweetness, hardness, blemishes) of each type of food product.

[0160]

[0184] In the mango example, if the wrinkle coverage score is 80% coverage (regardless of the value assigned to the color score) and the saleability threshold for mangoes is 65%, a binary decision can be made that the mango is not saleable. The overall quality metric can be a Boolean value such as whether the food product analyzed is saleable or not saleable.

[0161]

[0185] Any combination of rules and / or thresholds may be used to determine the overall quality metric of the food being analyzed, and the combination of rules and / or thresholds can vary based on the type of food being analyzed and the consumer and / or retailer's criteria for deeming the food being analyzed to be saleable, unsaleable, edible, inedible, good quality, poor quality, and / or acceptable quality. Thus, each type of food that can be analyzed using the disclosed techniques may have a differently defined set of characteristics / attributes that are used to determine the overall quality metric of that food.

[0162]

[0186] As alluded to above, the disclosed techniques provide for the establishment of a quality metric scale for different types of food products, regardless of how retailers and consumers, respectively, define the quality of the food products and how retailers and consumers may define such quality metrics. Thus, the disclosed techniques provide for the establishment of a scale that quantifies the quality of different types of food products to satisfy consumer quality concerns.

[0163]

[0187] FIG. 7 is a conceptual diagram of an example photo box 700 used to identify food quality based on image data. The photo box 700 can receive a shallow bin 708 (e.g., a pallet, a batch) of produce 710A-N (e.g., avocados). A camera 704 located within the photo box 700 can capture image data of the produce 710A-N on the shallow bin 708. The image data can be transmitted via the network 180 to one or more of the computer system 150 (e.g., for processing), the user device 170 (e.g., for display), and / or the food quality data store 600 (e.g., for storage). The image data can be processed using techniques described herein to identify quality characteristics of each of the produce 710A-N. For example, one or more of the models described herein can be executed locally at the computer system 150 to process the image data and identify quality characteristics of the produce 710A-N. Output from this processing can be transmitted and / or uploaded to a cloud-based service, such as the food quality data store 600. This output may then be retrieved by user device 170 and presented to relevant stakeholders, such as scientists or other users, who assess the quality of the produce 710A-N at storage facilities and / or packing houses.

[0164]

[0188] The photo box 700 may include walls on each side, a ceiling, and a flap 712 along one of the walls to allow the shallow box 708 to be placed inside the photo box 700 and removed from the photo box 700 once image capture is complete. The flap 712 may seal or otherwise cover an opening in the wall to prevent ambient or outside light from entering the interior of the photo box 700.

[0165]

[0189] The camera 704 can be located along the camera rig 702. The camera rig 702 can be configured on the ceiling of the photo box 700. The camera rig 702 can be made of aluminum tubes (e.g., three aluminum tubes) and a T-connector. The camera 704 can be attached to any of the aluminum tubes via a camera mount. One of the tubes can extend from the T-connector to a first wall of the photo box 700, a second tube can extend from the T-connector to a second wall opposite the first wall, and a third tube can extend from the T-connector to a third wall perpendicular to the first and second walls.

[0166]

[0190] Thus, the camera 704 can be trained downward. The camera 704 can be centered along the camera rig 702 inside the photo box 700. Sometimes the camera 704 can be off-center along the camera rig 702. In some implementations, the photo box 700 can include multiple cameras. For example, additional cameras can be positioned along the camera rig 702 extending between any walls of the photo box 700. In some implementations, additional cameras can be positioned along walls inside the photo box 700, rather than being attached to the camera rig 702 at the ceiling of the photo box 700. As a result, the shallow box 708 can be imaged from a variety of different angles.

[0167]

[0191] The camera 704 can be any type of imaging device described herein, including an RGB camera, a hyperspectral imaging device, a multispectral imaging device, a thermal imaging device, and / or any combination thereof. In some implementations, the camera 704 can be part of a magnetic resonance imaging (MRI) scanning system and / or a nuclear magnetic resonance (NMR) imaging system.

[0168]

[0192] The camera 704 may also be configured to one or more predefined settings. As an illustrative example, the camera 704 may be set to manual mode with a shutter speed of 1 / 20, F-stop F18, lens zoom 18mm, and dimmer set to maximum. Additionally, the image size may be large, the white balance may be set to direct sunlight, active D-lighting may be set to off, the ISO sensitivity may be 200, the autofocus may be set to auto-area autofocus, the flash may be off, image rotation may be off, HDR may be off, long exposure NR may be off, high ISO NR may be off, and vignette control may be normal. One or more other camera and capture settings may vary based on the setup, configuration, and / or components of the photo box 700.

[0169]

[0193] One or more lights 706A-N may be positioned throughout the photo box 700 to illuminate the shallow box 708. The lights 706A-N may provide uniform lighting inside the photo box 700 for better image capture by the camera 704. The lights 706A-N may be LEDs. In some implementations, the lights 706A and 706C may be located along / attached to the camera rig 702 to provide uniform lighting from the ceiling of the photo box 700. In some implementations, lights such as lights 706B and 706N may be located along any wall of the photo box 700. One or more other lighting configurations are also possible. Additionally, in some implementations, a diffuser may be located along / attached to the camera rig 702. A plastic diffuser may be placed along the length of the ceiling of the photo box 700 with an opening designated for the camera 704. A fabric diffuser, such as cloth, may be used with an opening designated for the camera 704 and placed along the length of the ceiling of the photo box 700.

[0170]

[0194] The photo box 700 can be located in a storage facility, packing house, warehouse, or other storage environment where produce 710A-N can arrive and be stored until shipped to a customer. One or more shallow bins for each category of processed produce (e.g., produce coated with a shelf-life extending coating solution) and unprocessed produce (e.g., produce not coated with any type of shelf-life extending coating solution) can be placed in the photo box 700 and imaged, one at a time. These shallow bins can then be displayed, imaged, and processed to identify quality changes that the processed and unprocessed produce may be subject to. The shallow bins can be placed in the photo box 700 by a human operator. The shallow bins may be automatically directed into the photo box 700 by a conveyor belt. For example, the photo boxes 700 can be in a line in a storage facility, thereby allowing for more automated, real-time analysis of the quality of the produce as it progresses through the storage facility.

[0171]

[0195] 8A and 8B illustrate a lime color analysis 800 using image data 802. Referring to FIG. 8A, image data 802 of a shallow bin (e.g., batch, palette) of limes can be captured using any of the techniques described herein. The image data 802 can be a hyperspectral image. The image data 802 can then be processed. For example, the image data 802 can be cropped (804) such that only shallow bins with limes are shown. An object detection model (see, for example, object detector model 902 of FIG. 9) can be applied to the cropped image data to detect limes 806. A bounding box, for example, can be determined for each lime. In some implementations, detecting limes 806 can also include positively identifying the type of produce within each bounding box. Once each lime is detected (806) in the cropped image data, a color analysis model (see, for example, color analyzer model 904 of FIG. 9, lime yellowing analyzer 614 of FIG. 6, etc.) can be applied to each bounding box 808. The color analysis model can output a numerical value indicative of a quality level of each lime in the image data 802. In some implementations, a general color analysis model may be applied, such as the color analyzer model 904. In some implementations, a lime color analysis model may be applied, such as the lime yellowing analyzer 614.

[0172]

[0196] Here, limes assigned values ​​of 1 to 9 appear more green, which is a more preferred color. Limes assigned values ​​of 10 to 22 appear more yellow, which is a less preferred color. Limes assigned values ​​of 23 to 33 appear dark yellow or brown, which is the least preferred color. For other shallow box limes, the values ​​assigned can vary within ranges and quality designations based on color.

[0173]

[0197] FIG. 8B illustrates another lime quality analysis output 810 that may be based on the color analysis 808 of FIG. 8A. As will be further described with reference to FIG. 17, the output 810 may display a color-coded bounding box around each identified lime. Limes of good quality (e.g., based on having a color within a desired threshold range) may be surrounded by a green bounding box, while limes of poor or bad quality (e.g., based on having a color not within a desired threshold range) may be surrounded by a red bounding box. The output 810 may also include an indication for each bounding box of whether the lime is of good or poor quality. One or more quality characteristics identified based on applying the color analysis 808 to the image data 802 may also be output with each bounding box. For example, a lime surrounded by a green bounding box and having an indication of "good" may still have some quality characteristics identified and thus may be output in the output 810, such as "Y1" representing yellowing severity 1 and "Y2" representing yellowing severity 2. Limes surrounded by a red bounding box and having a "poor" designation may have one or more quality characteristics, such as "Y1", "Y2", "Y3" indicating yellowing severity level 3, "DS" indicating dryness, and "RT" indicating spoilage, identified and therefore output at output 810. One or more other quality characteristics may be presented at output 810 as described throughout this disclosure.

[0174]

[0198] 9 illustrates a color analysis 900 of produce using image data. The color analysis 900 may be applied to a variety of different types of produce, including, but not limited to, apples, limes, avocados, zucchinis, and cucumbers. The color analysis 900 may be performed by running one or more models sequentially. In some implementations, the color analysis 900 may be performed by running models in parallel.

[0175]

[0199] Here, the object detector model 902 may be implemented in combination with the color analyzer model 904. In some implementations, both the object detector model 902 and the color analyzer model 904 may be trained to analyze the color of any type of product. Sometimes, the object detector model 902 may be trained to identify any type of product and the color analyzer model 904 may be trained to identify the quality of a particular type of product. Sometimes, the object detector model 902 may be trained to identify a particular type of product and the color analyzer model 904 may be trained to identify the quality of any type of product. Further, in some implementations, the object detector model 902 may be trained to identify a particular type of product and the color analyzer model 904 may be trained to identify the quality of a particular type of product.

[0176]

[0200] The object detector model 902 can be trained to identify any type of artifact using deep learning techniques such as CNN. As an example, the model 902 can be trained using approximately 200,000 images with 80 different classes of artifacts. A region proposal network (RPN) may be used to train the model to simultaneously predict a bounding box around each artifact in the image and assign a confidence score to each predicted bounding box. A region-based convolutional neural network (R-CNN) may be used to extract the predicted bounding boxes and then perform forward propagation on each predicted bounding box to detect and extract the artifact represented within the bounding box. In some implementations, the object detector model 902 may be trained to positively identify a particular type of artifact within each bounding box. In some implementations, one or more other image classification models may be applied to identify a particular type of artifact within each bounding box.

[0177]

[0201] Still referring to color analysis 900, an image 906 of a produce can be captured using the techniques described herein. An object detector model 902 and a color analyzer model 904 can be applied to the image 906 to generate an output 908. The output 908 can indicate the discoloration of the produce over time. An untreated produce 912 (e.g., without a coating or similar shelf-life extending coating solution) discolors faster and in a shorter time than a treated produce 910 (e.g., with a coating or similar shelf-life extending coating solution). Thus, the techniques described herein can be used to track the quality of the produce over multiple days. The techniques described herein also confirm that coating the produce with a shelf-life extending coating solution can slow the discoloration of the produce, as shown with treated produce 910, thereby maintaining a higher quality for a longer period of time.

[0178]

[0202] 10 illustrates a size distribution analysis 1000 using image data. The size distribution analysis 1000 may be applied to a variety of different types of produce, including, but not limited to, apples, limes, avocados, zucchinis, and cucumbers. The size distribution analysis 1000 may be performed by running one or more models sequentially. In some implementations, the size distribution analysis 1000 may be performed by running models in parallel.

[0179]

[0203] Here, an object detector model 902, a color analyzer model 904, and a size analyzer model 1002 can be applied to the image data 1004. The output 908 from the color analyzer model 904 can be used in combination with the output 1006 from the size analyzer model 1002 to determine overall quality information for the processed product 1008 and the unprocessed product 1010. For example, the processed product 1008 may age slower and may be of a preferred or desired size. Thus, the processed product 1008 may be of better quality, on average, than the unprocessed product 1010.

[0180]

[0204] As described herein, the size analyzer model 1002 can be trained to predict the size of each product imaged in the image data 1004 (see, e.g., FIG. 6). The model 1002 can also be trained to classify the products based on the predicted size. The model 1002 can also be trained to determine how many of the products are classified as each predicted size over time.

[0181]

[0205] The size analyzer model 1002 may output a bounding box around each identified product in the image data 1004. The model 1002 may also identify a numerical value indicating the predicted size of the product within each bounding box. The numerical value may then be output, for example, within each product's bounding box. In the image data 1004, the size analyzer model 1002 identified products having the following sizes: 48, 60, and 70.

[0182]

[0206] The size analyzer model 1002 can also generate an output 1006 that is a histogram representing the size distribution. The x-axis of the histogram represents product size. The y-axis of the histogram represents weekly inputs. Each bar in the histogram represents a total load. Thus, as shown in output 1006, approximately 12 loads per week have loads in the approximate size range of 49 to 55, while approximately 1 container per week has loads of 33.

[0183]

[0207] 11 illustrates a hardness analysis 1100 using hyperspectral imagery (HSI) images. The hardness analysis 1100 can be applied to a variety of different types of produce, including, but not limited to, apples, limes, avocados, zucchinis, and cucumbers. The hardness analysis 1100 can be performed by running one or more models sequentially. In some implementations, the hardness analysis 1100 can be performed by running models in parallel.

[0184]

[0208] Here, the object detector model 902 may be applied in combination with a hardness analyzer model 1102. See FIGS. 12 and 13 for further discussion of training the hardness analyzer model 1102 (e.g., the ripeness prediction model in FIG. 12). The produce may be imaged using an imaging device such as a hyperspectral imaging camera. The object detector model 902 and the hardness analyzer model 1102 may be applied to the image data 1104 to generate a hardness output 1106. As described further below with reference to FIGS. 12 and 13, the hardness output 1106 may provide a color indication (e.g., a color-coded bounding box) and a numerical value indicative of the overall hardness and / or ripeness identified for each produce. A hardness distribution output 1108 may also be generated. The hardness distribution output 1108 may be a histogram showing the distribution of hardness identified for all the produce in the image data 1104. The output 1108 may be color-coded to indicate different levels of hardness identified for a particular bin / batch of produce in the image data 1104. 11, avocados with an identified hardness value less than 15 are identified as very ripe, avocados with an identified hardness value between 15 and 29 are identified as medium ripe (e.g., perfect / ideal ripe), and avocados with an identified hardness value greater than 29 are identified as hard. Output 1108 may show hardness on the x-axis and frequency (e.g., how many avocados in a bin / batch are identified as being within a particular hardness value and / or hardness value range) on the y-axis.

[0185]

[0209] FIG. 12 illustrates an example avocado firmness prediction 1200 using HSI analysis. As described throughout this disclosure, hyperspectral images of produce such as avocados can be captured and used to identify features invisible to the human eye, such as firmness. Models can be trained using techniques described herein to infer firmness from hyperspectral images. Hyperspectral images include pixels with vectors that indicate the spectrum of a particular pixel. Thus, hyperspectral images can provide broader spectral information about a produce than other image data. This spectral information can vary based on the characteristics of the produce. For example, if an avocado has a blemish, a spectral difference can be visible in the avocado compared to an unblemished avocado. Models described herein can be trained to analyze the spectral information pixel by pixel to quantify quality features for the produce. As an example, a model can be trained to identify blemishes by detecting intensity levels in a spectral domain. Blemishes can be identified, found, and classified, for example, when the intensity levels in a spectral domain are below a threshold range. As another example, a model can be trained to identify dry matter content by detecting intensity levels in a region of the spectrum.

[0186]

[0210] Referring to firmness prediction 1200 of FIG. 12, an imaging device as described herein may be used to capture a hyperspectral image 1202 of a shallow box (e.g., a pallet, a batch) of avocados. For training purposes, destructive firmness measurements may also be taken of some or all of the avocados in the shallow box by piercing the skins of the avocados and / or crushing the avocados with a firmness measurement device. An average spectrum may be extracted from portions of the hyperspectral image 1202. For example, an object detection model may be applied to the hyperspectral image 1202 to generate a bounding box for each avocado. An average spectrum may then be extracted from each bounding box (e.g., for each avocado identified in the hyperspectral image 1202). A ripeness prediction model may then be trained to predict a state, such as firmness and / or ripeness, of each avocado based on the average spectrum and the destructive firmness measurements in each bounding box. Thus, the ripeness prediction model may be trained to identify the state of the avocado on a pixel-by-pixel basis. The model takes the average of all pixels that represent an avocado in the bounding box to identify an overall firmness value for that avocado. The ripeness prediction model can then output a firmness value that indicates the overall ripeness of each produce. During run-time, no destructive measurements of the avocados in the shallow box need to be made. Instead, a hyperspectral image can be taken and the ripeness prediction model can be applied to the hyperspectral image to identify a firmness value for each avocado in the picture.

[0187]

[0211] In some implementations, the model can be run for each pixel separately, generating an avocado hardness image where each pixel value can represent the hardness value at that location on the avocado. An overall avocado hardness value can be derived from the per-pixel hardness values ​​by averaging the image (e.g., a matrix).

[0188]

[0212] The output 1204 from the ripeness prediction model may indicate the firmness value in a variety of ways. For example, the firmness value may be represented as a color-coded bounding box. In other words, the bounding box of each avocado may be represented in one or more colors that indicate the firmness value. A green bounding box may indicate that the avocado is harder than other avocados in the shallow box. A yellow bounding box may indicate that the avocado has a better firmness level than other avocados in the shallow box. A red bounding box may indicate that the avocado has a less firmness level than other avocados in the shallow box. In some implementations, the bounding box color may indicate firmness on a normalized scale that is applied to all shallow boxes of avocados imaged. One or more other indicia may be used to indicate the firmness value in the output 1204 (e.g., different color tones and / or patterns may be used to indicate each bounding box based on the identified firmness value).

[0189]

[0213] The output 1204 may also provide a numerical value indicating an aggregate firmness value for each avocado. The numerical value may be represented in the output 1204 as an overlay on the bounding box of the avocado. The numerical value may be a normalized scale value (e.g., 0 to 100) that is applied to all bins of the avocado imaged. The numerical value may be a scale based on a variety of factors including, but not limited to, origin, time of year, and other metadata. The numerical value may be a scale of values ​​and thus may be based on the distribution of firmness values ​​of avocados in a particular bin captured in the hyperspectral image 1202.

[0190]

[0214] In the example output 1204, avocados with a firmness value between 5 and 15 are shown with a red bounding box, meaning that these avocados are identified as having a low firmness or higher ripeness. Avocados with a firmness value between 16 and 29 are shown with a yellow bounding box, meaning that these avocados are identified as having a good firmness level or ideal / optimal ripeness. Avocados with a firmness value between 30 and 36 are shown with a green bounding box, meaning that these avocados are identified as being lower firmness or lower ripeness.

[0191]

[0215] Output 1204 can be used (e.g., by computer system 150) to predict the ripeness of avocados over one or more days. For example, time lapse 1206 shows images of avocados in hyperspectral image 1202. Time lapse 1206 is taken on day 1, when output 1204 is generated. Each avocado is surrounded by a color-coded bounding box that indicates the predicted firmness / ripeness of that avocado using the techniques described herein. As the days pass, hyperspectral images of the avocado casks can be captured and a ripeness prediction model can be applied to non-invasively and non-destructively measure and predict ripeness. Time lapse 1208 shows avocados on day 5. As shown, avocados that were within the red bounding box on day 1 now appear darker on day 5, indicating that such avocados are of lower firmness and are more ripe. The color of the avocado changes as predicted by applying the ripeness prediction model to the hyperspectral image 1202 on day 1. Thus, the ripeness prediction model can be useful in accurately predicting avocado firmness and ripeness without having to destructively measure the avocado's firmness.

[0192]

[0216] FIG. 13 illustrates another example avocado firmness prediction 1300 using HSI analysis. As described throughout this disclosure, hyperspectral images of produce such as avocados can be captured and used to identify features invisible to the human eye, such as firmness. A model can be trained using techniques described herein to infer firmness from hyperspectral images. An image 1302 of the avocado can be generated as described throughout this disclosure. For example, an image captured by an HSI camera can be used to obtain HSI data. An image 1302 can then be generated from the HSI data. The firmness of the avocado can be inferred by applying one or more machine learning trained models to the HSI data of the avocado (e.g., Xshore). As described above, each identified avocado can be assessed for firmness using one or more models. Output from the model application can include a numerical value, other metrics, and / or color (e.g., or other indicia such as pattern, shape, tone, etc.) of each avocado.

[0193]

[0217] In FIG. 13, the estimated avocado firmness output 1304 shows a color bounding box and a numerical value for each avocado in the image 1302, as identified by applying one or more models described herein. Avocados found in a green bounding box may be identified as firm. Avocados found in a yellow bounding box may be identified as optimal firmness. Avocados found in red may be identified as ripe. Each avocado is also assigned a numerical value that represents the avocado's firmness level. The numerical value may be within a range. The range may be predetermined. The range may be relative, such as based on the firmness level identified for each avocado in the image 1302. As shown in output 1304, avocados having a hardness level between 86 and 107 are identified as hard (within a green bounding box), avocados having a hardness level between 57 and 78 are identified as optimal (within a yellow bounding box), and avocados having a hardness level of 55 are identified as ripe (within a red bounding box).

[0194]

[0218] One or more of the models may also output a histogram 1306 showing the distribution of hardness levels of all the avocados in the image 1302. The X-axis of the histogram 1306 represents hardness (Xshore) on a scale of 1 to 110. One or more other scales may be used. For example, the scale may be a predefined general scale used to quantify the hardness of a particular type of produce. The scale may be relative and dynamic based on the hardness levels identified for all the avocados in the image 1302. The Y-axis of the histogram 1306 represents frequency on a scale of 0 to 8. As mentioned above, the scale may vary based on pre-specified information and / or hardness analysis for all the avocados in the image 1302. The frequency may indicate how many avocados in the batch have each of the estimated hardness levels of optimal, hard, and ripe.

[0195]

[0219] In the example histogram 1306, avocados identified as ripe (e.g., having a hardness of up to 60) have a frequency of 6. Avocados identified as optimal (e.g., having a hardness of 60-85) can have a variety of frequencies. For example, avocados with a hardness of 60-65 can have a frequency of 8, avocados with a hardness of 65-70 can have a frequency of 5, avocados with a hardness of 70-75 can have a frequency of 3, and avocados with a hardness of 75-85 can have a frequency of 1. Avocados identified as hard (e.g., having a hardness of 92-108) can also have a variety of frequencies. For example, avocados with a hardness of 92-95 can have a frequency of 2, and avocados with a hardness of 95-108 can have a frequency of 1. As mentioned above, the distribution of hardness levels can vary for each bin, pallet, and / or batch of avocados. Additionally, although Figure 13 is described with reference to avocados, the techniques, processes, and analyses described herein can also be applied to other types of produce, such as those described throughout this disclosure.

[0196]

[0220] FIG. 14 illustrates a per-pixel avocado firmness prediction 1400 in HSI data. A hyperspectral image of a produce such as an avocado can be captured and used to identify features invisible to the human eye, such as firmness. One or more models may be used and trained to identify firmness from the hyperspectral image. As a result, destructive techniques such as piercing and / or squashing the produce may not be necessary to accurately identify the firmness of the produce. The models can be trained to infer firmness for each pixel as shown in the hyperspectral image data 1402. Pixels that appear brighter (e.g., less transparent, more opaque, more luminosity) may represent a higher firmness in the avocado than pixels that appear less bright (e.g., more transparent, less opaque, less luminosity). The pixels may be represented with one or more other indicia to indicate the firmness level. For example, pixels representing firmness levels above a predetermined threshold range may be represented with a first color, such as yellow. Pixels representing firmness levels within the threshold range may be represented with a second color, such as green. Pixels representing hardness levels below the threshold range may be represented by a third color, such as blue. Various shades and / or degrees of color or other indicia (e.g., patterns, shades, etc.) may be used to output an inferred hardness level of the avocado. A model may be trained to identify an aggregate hardness level for each avocado based on an analysis of the pixels representing the avocado.

[0197]

[0221] 14 also shows an RGB image 1404 of the same group of avocados shown by the hyperspectral image data 1402. As shown here, the RGB image 1404 merely shows the outside of the avocado as it is visible to the human eye. From the RGB image 1404, it is not possible for a person to know the hardness. Thus, to infer the avocado's hardness, a person would need to crush, pierce, or perform some other destructive technique on the avocado. Thus, the hyperspectral image data 1402 can provide a deeper analysis of the quality attributes of the avocado that may not be visible to the human eye.

[0198]

[0222] Although FIG. 14 is described with reference to avocados, the same techniques, processes, and analyses can be performed on other types of produce as described throughout this disclosure.

[0199]

[0223] FIG. 15 illustrates stem die analysis 1500 and 1510 using HSI data. Hyperspectral images of produce such as avocados can be captured and used to identify features invisible to the human eye, such as stem die. One or more models can be used and trained to identify stem die from hyperspectral images. The models can be trained, for example, to detect and visualize the onset of stem die in each imaged produce. In the example avocado analysis 1500, which can be performed on day 0, an avocado such as avocado 1502 is shown with more spots than an avocado such as avocado 1504. The spots in avocado 1502 can be visualized using indicia such as a color and / or pattern (e.g., bright yellow). For example, the spots can be visualized in yellow and / or green. The spots represent stem die. Thus, avocado 1502 has much more stem die than avocado 1504, even though avocado 1502 may be a moderate case of stem die that may appear within a given noise level. On the other hand, avocado 1504 has fewer, if any, spots. Some of the spots on avocado 1504 are more transparent (e.g., less opaque, less bright, etc.) than more extensive spots on other avocados, such as avocado 1502. In other words, analysis 1500 may return images showing various levels / degrees of stem blight on each avocado. One or more models may be trained using this images to identify an overall quality metric for each avocado. An avocado with lighter, more opaque, and / or colored spots near the stem of the avocado may have more severe stem blight (and thus be classified as lower quality) than an avocado with no spots or spots near the stem that are less bright, less opaque, and / or have no colored markings indicative of stem blight (and thus be classified as higher quality).

[0200]

[0224] Example analysis 1510 visualizes the same avocado as analysis 1500. However, analysis 1510 shows the avocado on day 7 of imaging. In this example, avocado 1502 has significantly stronger signs of stem dieback near the stem, represented by a lighter, more opaque yellow color than day 0 in analysis 1500. Additionally, avocado 1504 shows little sign of stem dieback in analysis 1500 on day 0, whereas in analysis 1510 on day 7, avocado 1504 has also begun to develop some stem dieback.

[0201]

[0225] FIG. 16 illustrates an example aging analysis 1600 using HSI data. Hyperspectral images of produce, such as avocados, can be captured over time and used to identify features invisible to the human eye. One of those features is the aging of the produce. An RGB image 1602 can show an avocado on day 1 as green. Compressed HSI data 1604 of the same avocado on day 1 can show additional information about the spectrum that the human eye cannot discern. One or more models described herein can analyze a spectrum 1606 from the compressed HSI data 1604 to identify ripeness changes in the avocado, and therefore, aging of the avocado. The models can be trained to identify changes in the invisible IR range of the spectral data 1606, which can indicate that chlorophyll is degrading and the avocado is aging. An RGB image 1608 shows the same avocado on day 7. Compressed HSI data 1610 shows additional information about the spectrum that the human eye cannot discern. As the avocado aged from day 1 to day 7, the skin of the avocado became more wrinkled, the size / volume of the avocado decreased, and one or more visual characteristics of the avocado may have changed, as shown by a comparison of the compressed HSI data 1604 to the compressed HSI data 1610. The spectrum 1612 may be analyzed to identify changes in ripeness of the avocado on day 7, which may then be used to identify senescence of the avocado. Although FIG. 16 is described with reference to avocados, the senescence analysis 1600 using HSI data may be performed on any other type of produce described throughout this disclosure.

[0202]

[0226] FIG. 17 shows an example output 1700 from an internal quality analysis of an avocado. As described throughout this document, some produce may be cut open and then imaged to assess the quality of a batch of produce. The cut open produce may be placed in a shallow bin, on a pallet, or otherwise located inside a photo box, as described with reference to FIG. 7. An image of the cut open produce may be captured. The image may be an RGB image and / or a hyperspectral image. The image may be any other type of image described throughout this disclosure. In some implementations, an image of the exterior of the produce may be captured. The produce may then be cut open so that the interior of the produce may also be imaged. The exterior and interior images may then be analyzed using techniques described herein to identify the quality of the produce.

[0203]

[0227] As shown in the example avocado internal quality analysis output 1700, a bounding box is identified around each avocado that is cut open. The bounding boxes can be colored or represented by different indicia (e.g., patterns such as dotted lines) based on the aggregate quality level identified for each avocado. For example, avocados with no internal defects or minimal internal defects overall can be surrounded by a green bounding box. These bounding boxes may include a metric that indicates the identified quality level. For example, the green bounding box can include a string value of "good". Avocados with some internal defects or an amount of internal defects within a threshold range can be surrounded by an orange bounding box. These bounding boxes can include a string value of "acceptable" indicating that the avocado has some internal defects that do not make it either good or bad. Avocados with internal defects or an amount of internal defects above a threshold range can be surrounded by a red bounding box. These bounding boxes may include a string value of "poor". One or more other metrics may be used to identify the overall quality of the product based on the detected internal defects.

[0204]

[0228] One or more models may be applied to the bounding boxes to determine what types of internal defects are present in each avocado and whether the avocado is good, acceptable, or poor. The identified types of internal defects may also be presented in output 1700. For example, some avocados that are overall "good" may still have some internal defects (e.g., internal defects below a threshold range). The bounding boxes of "good" quality avocados with some internal defects may be annotated with an abbreviation or other metric indicating the type of defect identified. Similarly, avocados that are overall "acceptable" or "poor" quality may also be annotated with an abbreviation or other metric indicating the type of defect identified. In the example output of FIG. 17, the abbreviations used may include "DB" for extensive browning, "VB" for vascular browning, "SR" for stem dieback, and "IR" for internal rot.

[0205]

[0229] Although FIG. 17 is described with reference to avocados, the same processes and techniques may be applied to other types of produce to achieve the same or similar results and internal quality analysis outputs.

[0206]

[0230] In some implementations, the internal quality of the produce can be assessed using non-destructive techniques. For example, a hyperspectral image of the exterior of the produce can be used to identify internal quality characteristics below the surface of the produce. As another example, an MRI can be used to visualize the interior of the produce. An MRI scanning system can be located in line (see, e.g., FIG. 1A) and / or instead of or in addition to a photo box 700 (see, e.g., FIG. 7). A shallow box or pallet of produce can be placed inside the MRI scanning system to generate an MRI of the produce. The MRI can capture and image the fibers within the produce, which can then be analyzed using one or more of the models described herein. As a result, internal defects such as rottenness or spoilage can be identified without destroying the produce. As yet another example, an NMR system can be used to non-destructively identify the internal quality of the produce.

[0207]

[0231] 18 is a block diagram of system components that can be used to implement a system for assessing the quality of one or more food products. Computing device 1800 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Mobile computing device is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementation of the invention described and / or claimed herein.

[0208]

[0232] The computing device 1800 includes a processor 1802, a memory 1804, a storage device 1806, a high-speed interface 1808 connecting to the memory 1804 and a number of high-speed expansion ports 1810, and a low-speed interface 1812 connecting to a low-speed expansion port 1814 and the storage device 1806. Each of the processor 1802, the memory 1804, the storage device 1806, the high-speed interface 1808, the high-speed expansion port 1810, and the low-speed interface 1812 are interconnected using various buses and may be mounted on a common motherboard or otherwise mounted as appropriate. The processor 1802 may process instructions for execution within the computing device 1800, including instructions stored in the memory 1804 or the storage device 1806 for displaying graphical information for a GUI on an external input / output device, such as a display 1816 coupled to the high-speed interface 1808. In other implementations, multiple processors and / or multiple buses may be used as appropriate, along with multiple memories and multiple types of memories. Additionally, multiple computing devices may be connected, each providing a portion of the required operations (eg, as a bank of servers, a collection of blade servers, or a multi-processor system).

[0209]

[0233] The memory 1804 stores information within the computing device 1800. In some implementations, the memory 1804 is one or more volatile memory units. In some implementations, the memory 1804 is one or more non-volatile memory units. The memory 1804 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0210]

[0234] The storage device 1806 can provide mass storage for the computing device 1800. In some implementations, the storage device 1806 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state memory device, or an array of devices including a storage area network or other configuration of devices. The computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The computer program product can be tangibly embodied in a computer or machine-readable medium, such as the memory 1804, the storage device 1806, or a memory on the processor 1802.

[0211]

[0235] The high-speed interface 1808 manages bandwidth-intensive operations for the computing device 1800, and the low-speed interface 1812 manages less bandwidth-intensive operations. Such an allocation of functions is merely an example. In some implementations, the high-speed interface 1808 is coupled to the memory 1804, the display 1816 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 1810, which can accept various expansion cards (not shown). In this implementation, the low-speed interface 1812 is coupled to the storage device 1806 and the low-speed expansion port 1814. The low-speed expansion port 1814 can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and can be coupled, for example, via a network adapter, to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or router.

[0212]

[0236] The computing device 1800 can be implemented in a number of different forms, as shown. For example, the computing device 1800 can be implemented as a standard server 1820 or multiple times within a cluster of such servers. In addition, the computing device 1800 can be implemented within a personal computer, such as a laptop computer 1822. The computing device 1800 can also be implemented as part of a rack server system 1824. Alternatively, components from the computing device 1800 can be combined with other components in a mobile device (not shown), such as a mobile computing device 1850. Each such device can include one or more of the computing device 1800 and the mobile computing device 1850, and the entire system can be comprised of multiple computing devices in communication with each other.

[0213]

[0237] The mobile computing device 1850 includes, among other components, a processor 1852, a memory 1864, input / output devices such as a display 1854, a communication interface 1866, and a transceiver 1868. The mobile computing device 1850 may also include a storage device, such as a microdrive or other device, to provide additional storage. Each of the processor 1852, memory 1864, display 1854, communication interface 1866, and transceiver 1868 are interconnected using various buses, and several of these components may be mounted on a common motherboard or in other suitable manners.

[0214]

[0238] The processor 1852 can execute instructions in the mobile computing device 1850, including instructions stored in the memory 1864. The processor 1852 can be implemented as a chipset of chips including separate analog and digital processors. The processor 1852 can provide for the cooperation of other components of the mobile computing device 1850, such as control of a user interface, applications executed by the mobile computing device 1850, and wireless communication by the mobile computing device 1850.

[0215]

[0239] The processor 1852 can communicate with a user through a control interface 1858 and a display interface 1856, which is coupled to a display 1854. The display 1854 can be, for example, a TFT (thin film transistor liquid crystal display) display or an OLED (organic light emitting diode) display or other suitable display technology. The display interface 1856 can include appropriate circuitry to drive the display 1854 to present graphical and other information to the user. The control interface 1858 can receive commands from a user and translate those commands for transmission to the processor 1852. Additionally, an external interface 1862 can provide communication with the processor 1852 to enable short-range communication between the mobile computing device 1850 and other devices. The external interface 1862 can be provided, for example, for wired communication in some implementations or for wireless communication in other implementations, and multiple interfaces can also be used.

[0216]

[0240] The memory 1864 stores information within the mobile computing device 1850. The memory 1864 may be implemented as one or more of one or more computer readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 1874 may also be provided and may be connected to the mobile computing device 1850 through an expansion interface 1872, which may include, for example, a SIMM (single in-line memory module) card interface. The expansion memory 1874 may provide extra storage space for the mobile computing device 1850 or may store applications or other information for the mobile computing device 1850. In particular, the expansion memory 1874 may include instructions for implementing or supplementing the processes described above and may also include secure information. Thus, for example, the expansion memory 1874 may provide a security module for the mobile computing device 1850 and may be programmed with instructions that enable secure use of the mobile computing device 1850. Additionally, secure applications can be provided via the SIMM card in a non-hackable manner, along with additional information such as location identification information on the SIMM card.

[0217]

[0241] The memory may include, for example, flash memory and / or NVRAM memory (non-volatile random access memory), as discussed below. In some implementations, the computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The computer program product may be a computer or machine readable medium, such as memory 1864, expansion memory 1874, or memory on processor 1852. In some implementations, the computer program product may be received in a propagated signal, for example, via transceiver 1868 or external interface 1862.

[0218]

[0242] The mobile computing device 1850 can communicate wirelessly through a communication interface 1866, which can include digital signal processing circuitry, if necessary. The communication interface 1866 can provide communication under various modes or protocols, such as GSM voice calls (Global System for Mobile Communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (Code Division Multiple Access), TDMA (Time Division Multiple Access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), among others. Such communication can be through a transceiver 1568 using, for example, radio frequencies. Additionally, short-range communication can be performed using, for example, Bluetooth, Wi-Fi, or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 1870 can provide additional navigation- and location-related radio data to the mobile computing device 1850, such data can be used as appropriate by applications executing on the mobile computing device 1850.

[0219]

[0243] The mobile computing device 1850 can also communicate audibly using a voice codec 1860, which can receive verbal information from a user and convert it into usable digital information. The voice codec 1860 can generate audible sounds for the user as well, such as through a speaker in a handset of the mobile computing device 1850. Such sounds can include sounds from a voice telephone call, can include recorded sounds (e.g., voice messages, music files, etc.), and can also include sounds generated by applications running on the mobile computing device 1850.

[0220]

[0244] The mobile computing device 1850 can be implemented in a number of different forms as shown in the figure. For example, the mobile computing device 1850 can be implemented as a cellular telephone 1880. The mobile computing device 1850 may also be implemented as part of a smartphone 1882, personal digital assistant, or other similar mobile device.

[0221]

[0245] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which can be special-purpose or general-purpose, and can be coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0222]

[0246] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or assembly / machine language. As used herein, the terms machine-readable medium and computer-readable medium refer to any computer program product, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term machine-readable signal refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0223]

[0247] To provide for user interaction, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) that allows the user to provide input to the computer. Other types of devices can be used to provide user interaction as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and can receive any form of input from the user, including acoustic, voice, or tactile input.

[0224]

[0248] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), middleware components (e.g., an application server), front-end components (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0225]

[0249] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0226]

[0250] FIG. 19 illustrates an example mango wrinkle analysis using image data 1900 and the wrinkle analyzer 670 described with reference to FIG. 6. Using the disclosed techniques, the image data 1900 can be processed 1902 to generate a grid 1904 (e.g., a mask) having patches. In the example of FIG. 19, the grid 1904 can include seven patches horizontally across the image data 1900 and eleven patches vertically down the image data 1900. The image data 1900 may be divided into any other number of patches in the grid 1904. For example, the image data 1900 may be divided into a larger number of patches in the grid 1904 to increase the accuracy and / or specificity of the total wrinkle coverage.

[0227]

[0251] A classifier 1906, such as a trained machine learning model or neural network, may then classify 1908 each patch in the grid 1904 to determine whether each patch represents a wrinkle. The classifier 1906 may be trained using training data having labels with annotations of wrinkles in the image patches. Thus, the classifier 1906 may be trained to perform a binary decision, such as whether each patch in the grid 1904 has a wrinkle or not. For example, the classifier 1906 may receive image patches 1912A and 1912N. The classifier 1906 may determine that the image patch 1912A includes a wrinkle and therefore assign a binary value 1914A of 1 to the image patch 1912A. On the other hand, the classifier 1906 may determine that the image patch 1912N does not include a wrinkle and therefore assign a binary value 1912B of 0 to the image patch 1912N.

[0228]

[0252] An output 1910 may be generated as a result of the classification in 1908. The output 1910 may include, for example, a percentage of wrinkle coverage (e.g., a wrinkle coverage score) for the entire mango represented in the image data 1900. The overall percentage of wrinkle coverage may be generated based on quantifying (e.g., summing, averaging, or otherwise aggregating) the binary values / identifications of all patches in the grid 1904. In some implementations, the overall percentage of wrinkle coverage may be determined based on summing all patches classified as wrinkled and dividing by the total number of patches in the grid 1904.

[0229]

[0253] In the example of Figure 19, approximately 5% of the surface of the mango was identified as being covered with wrinkles. In some implementations, the output 1910 may include an image 1900 having a grid 1904 and one or more patches within the grid 1904 that are annotated (e.g., highlighted, labeled) as being classified as having wrinkles. Here, four patches within the grid 1904 are annotated as being classified as having wrinkles, which collectively account for 5% of the surface of the mango.

[0230]

[0254] FIG. 20 illustrates an example strawberry calyx browning analysis using image data 2000 and the calyx browning analyzer 672 described with reference to FIG. 6. Using the disclosed techniques, strawberry image data 2000 can be processed to identify color values ​​and correlate such color values ​​with quality levels. For example, image data 2000 can be processed to extract specific color channels 2002, such as color channels A and B. At 2004 and / or 2006, a predetermined binary threshold can be applied to the generated image. A mask with the original RGB image can then be applied at 2008. Color values ​​can be identified and extracted from the generated image. From among the identified and extracted color values, a median color value can be identified (2010). The median color value can then be compared to one or more calyx browning thresholds to determine whether the strawberry represented in image data 2000 has calyx browning and / or whether the strawberry has a sufficient or threshold amount of calyx browning to reduce the overall quality of the strawberry. As described with reference to FIG. 6, the analyzer 672 can generate an overall quality score for the strawberry of “good,” “poor,” or “acceptable” based on comparing the calculated median color to the calyx browning threshold criteria.

[0231]

[0255] FIG. 21 illustrates an example banana stage analysis using image data and the banana color analyzer 674 described with reference to FIG. 6. Using the disclosed techniques, banana image data 2100 can be provided as input to a model 2102. The model 2102 can be a machine learning model and / or neural network trained to extract color values ​​(e.g., green areas, brown areas) in the floral design image data and classify the extracted color values ​​as color metrics that map to banana ripeness stages. The color values ​​can be extracted from the image data using any of the disclosed techniques. The extracted color values ​​can then be checked against threshold color value ranges, each corresponding to a different ripeness stage associated with the banana (e.g., color metric-to-stage mapping graph 2106). The model 2102 then generates an output 2104 indicating the banana ripeness stage based on the banana's color. In the example of FIG. 21, image data 2100 may be analyzed by analyzer 674 using model 2102 to determine that the banana in image data 2100 is at stage 5 of banana ripeness (e.g., on a scale of color stages from 1 to 7, where stage 1 is the lowest ripeness or before the ripening process has begun and stage 7 is the highest ripeness).

[0232]

[0256] 22A and 22B show an example cherry stem color analysis using image data and the cherry stem color analyzer 676 described with reference to FIG. 6. Referring to both FIG. 22A and FIG. 22B, using the disclosed techniques, an object detection technique 2200 can be applied to the image data to identify cherries in the image data. A bounding box 2202 can be generated around the identified cherries. A stem color analysis 2204 can be performed to extract or otherwise separate the stems from the bodies of the cherries within the bounding box 2202. For example, the stems can be separated using machine learning techniques or machine learning trained models, as described with reference to the analyzer 676 of FIG. 6. The separated stems can be masked with the source image for color quantification. The stem color analysis 2206 can be followed by identifying and extracting one or more color channels within the bounding box 2202, and then applying a predetermined binary threshold to the bounding box 2202. As a result, color values ​​of the isolated stems can be extracted and compared to one or more threshold color ranges to determine whether the stem color (or stem median, average, calculated mean, or other statistically average color value) corresponds to cherries of "good" quality, cherries of "acceptable" quality, or cherries of "poor" quality. As shown in the color analysis output 2208, a threshold amount of hue or color value can be indicative of "good" quality. As described herein, other threshold amounts of hue or color value may be defined and associated with one or more different quality levels of cherries.

[0233]

[0257] Figure 23 is a flow chart of a process 2300 for determining an overall quality metric of a food product. As described with reference to Figure 6, the overall quality metric may be determined using nested rules that are applied to quality determinations made by different quality assessment analyzers to classify the overall quality of the food product. Process 2300 may be performed by computer system 150. Process 2300 may also be performed by one or more other computing systems, devices, and / or servers.

[0234]

[0258] Referring to process 2300 of FIG. 23, the computer system can access quality characteristics A-N of the food product in block 2302. The quality characteristics A-N can be quality scores or other outputs generated by one or more quality assessment analyzers applied to the image data of the food product. See FIG. 6 for further discussion of the quality assessment analyzers and the quality characteristics A-N each analyzer may generate. The computer system can poll the quality assessment analyzers for each quality characteristic A-N. The computer system can also receive the quality characteristics A-N from the analyzers as the characteristics are made and / or in batches or at predetermined time intervals. In some implementations, the quality characteristics A-N can be stored in a data store and then retrieved by the computer system in block 2302.

[0235]

[0259] In block 2304, the computer system can access rule-based mappings of different ranges of values ​​of quality attributes A-N (including thresholds in addition or alternatively) to different enumerated categories of food quality. The computer system can access the rule-based mappings from a data store. The computer system can determine which rule-based mapping to access based on the type of food being analyzed and / or the quality attributes A-N accessed in block 2302. The range of values ​​of the quality attributes A-N can vary based on the type of food. Additionally or alternatively, the range of values ​​of each quality attribute A-N can vary based on the type of quality assessment performed to generate the quality attribute. Furthermore, each food can have different rule-based mappings based on various characteristics of the food, the food type, and the quality assessment used to assess the quality of the food.

[0236]

[0260] The computer system iteratively determines whether each rule-based mapping is satisfied for each quality specification A-N (block 2306). The computer system identifies an enumerated category of food quality based on the determination that each rule-based mapping is satisfied (block 2308). For example, if each rule is satisfied for one of the quality specifications A-N, the computer system can identify a category corresponding to the determination that the rule is satisfied for that particular quality specification. The identified category can indicate, for example, a saleable or edible food. On the other hand, if no rule is satisfied for a quality specification, the computer system can continue iteratively for each rule-based mapping for that particular quality specification. If none of the rules are satisfied for a quality specification, the computer system can identify a category that is unsalable or inedible. The computer system can iteratively determine whether each rule for any remaining quality specifications A-N is satisfied.

[0237]

[0261] The computer system may assign the identified category to the food product as an overall quality metric for the food product in block 2310. The computer system may then return an overall quality metric for the food product in block 2312, as described throughout this disclosure.

[0238]

[0262] As an illustrative example, for a mango, the computer system may access two quality characteristics at block 2302: wrinkle score and color score. At block 2304, the computer system may identify and retrieve a rule-based mapping that corresponds to the wrinkle score and color score of the mango. The rule-based mapping for the wrinkle score of the mango may have different rules and / or thresholds that must be met to classify the food product into one or more different buckets or categories of quality, while the rule-based mapping for the color score of the mango may have other rules and / or thresholds that must be met to classify the food product into one or more buckets or categories of quality. The computer system may determine at block 2306 that the wrinkle score is within a threshold range of values ​​that correspond to the listed categories of saleable or edible. As a result, the computer system may identify the saleable or edible category at block 2308 and assign that category as the overall quality metric for the food product at block 2310. An indication that a particular mango is saleable or edible may be returned to the associated user's computing device, for example, in a GUI display, in block 2312. The associated user may then use this indication to determine whether to transport the mango to a retail store for sale, where to place the mango on the retail store's shelves for sale, whether to sell the mango at a higher or discounted price, and / or one or more other actions in the mango's supply chain.

[0239]

[0263] Although the specification contains many specific implementation details, these should not be construed as limitations to the scope of the disclosed technology or limitations to what may be claimed, but rather as descriptions of features that may be specific to certain embodiments of the disclosed technology. Certain features described herein with respect to separate embodiments may be implemented in combination, either partially or in whole, in one embodiment. Conversely, various features described with respect to one embodiment may be implemented separately in multiple embodiments or in any suitable subcombination. Furthermore, although features may be described herein and / or initially claimed as operating in a particular combination, one or more features from the claimed combination may, in some cases, be removed from that combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination. Similarly, although operations may be described in a particular order, this should not be understood as requiring that such operations be performed in that particular order or sequentially, or that all operations be performed, to achieve desired results. Certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims.

Claims

1. 1. A method for identifying a quality level of a food product using image data, comprising: receiving, by a computing system, image data of the food product from the imaging device; performing, with the computing system, object detection on the image data to identify a bounding box around each of the food items in the image data; identifying, by the computing system, a grid structure for the image data based on the bounding boxes around each of the food items in the image data, wherein each bounding box is assigned a grid index in the grid structure, and the grid index is used to identify the food item in a data store; identifying, by the computing system, for each of the food products, a quality level of the food product by applying a plurality of trained models to the bounding box portions of the image data that include each of the food products, each of the trained models being trained using image training data of other foods, the image training data being annotated based on a previous identification of a first portion of the other food product as having a poor quality feature and a previous identification of a second portion of the other food product as having a good quality feature, the other foods being of the same food type as the food product, and the plurality of trained models including a first trained model trained to identify a first quality feature and a second trained model trained to identify a second quality feature that is different from the first quality feature; determining, by the computing system, for each of the food products, a quality level score for the food product based on the identified quality level of the food product; storing, by the computing system, for each of the food items in the data store: (i) the bounding box portion of the image data that includes the food item; (ii) the grid index; (iii) the identified quality level of the food item; and (iv) the determined quality level score of the food item; A method comprising:

2. 10. The method of claim 1, further comprising transmitting, by the computing system to a user computing device, the quality level scores of the food items in the image data for display on the user computing device.

3. retrieving, by the computing system, for each of the food products, the quality level score of the food product from the data store; identifying, by the computing system, supply chain information for the food product, including an existing supply chain schedule and destination for the food product; determining, by the computing system, whether to modify the supply chain information of the food product based on the retrieved quality level score; generating, by the computing system, modified supply chain information based on the received quality level scores in response to a determination to modify the supply chain information, the modified supply chain information including one or more of a modified supply chain schedule and a modified destination for the food product; transmitting, by the computing system, the modified supply chain information to one or more supply chain participants to implement the modified supply chain information; The method of claim 1 further comprising:

4. 4. The method of claim 3, wherein the modified supply chain information includes instructions that, when executed by the one or more supply chain participants, move the food product for outbound shipment to an end consumer that is geographically closest to the location of the food product.

5. 4. The method of claim 3, wherein the modified supply chain information includes instructions that, when executed by the one or more supply chain participants, cause at least one of: (i) moving the food product for outbound shipment to a food processing plant; (ii) changing controlled atmospheric conditions surrounding the food product; (iii) changing ripening conditions for the food product; (iv) changing cold storage conditions for the food product; or (v) moving the food product for outbound shipment to an end consumer that is geographically closest to the location of the food product.

6. 6. The method of claim 1, wherein each of the plurality of trained models comprises one or more layers having (i) image training data of the other foods and (ii) labels indicative of a food quality classification of each of the other foods depicted in the image training data.

7. The method of any one of claims 1 to 5, wherein the food product is at least one of avocado, lime, lemon, apple, berry, and mango.

8. identifying, by the computing system, the type of food depicted in the image data using object recognition; selecting, by the computing system, one or more of the plurality of trained models to apply to the bounding box portion of the image data based on the identified type of food, wherein the selected trained models are trained to identify quality features of food of the same type; The method of any one of claims 1 to 5, further comprising:

9. and identifying, by the computing system, for each of the food products, a quality level of the food product. calibrating the color of the image data to maintain color consistency in the image data; determining a median Euclidean distance between a color of the food product within the bounding box portion of the image data and a reference color of the food product based on application of a color analyzer trained model to the calibrated bounding box portion of the image data, the reference color being associated with a preferred quality level of the food product; determining whether the median Euclidean distance exceeds a threshold; identifying the quality level of the food product as poor quality in response to determining that the median Euclidean distance exceeds the threshold; identifying the quality level of the food product as good quality in response to determining that the median Euclidean distance is less than the threshold; and The method according to any one of claims 1 to 5, comprising:

10. 6. The method of claim 1, wherein identifying, by the computing system, for each of the food products, the food product quality level comprises applying hyperspectral cube processing to the bounding box portion of the image data that includes each of the food products.

11. and identifying, by the computing system, for each of the food products, a quality level of the food product. determining the extent to which the food products have yellowed based on applying an apple yellowing trained model to the bounding box portions of the image data that include each of the food products; and assigning the quality level score to the food product based on the extent to which the food product has yellowed, wherein a score of 0 or greater but less than a threshold level indicates (i) good quality and (ii) no yellowing, and a score of 100 or less but greater than the threshold level indicates (i) poor quality and (ii) maximum yellowing of the food product; The method according to any one of claims 1 to 5, comprising:

12. and identifying, by the computing system, for each of the food products, a quality level of the food product. determining the extent to which the food products have yellowed based on applying a lime yellowing trained model to the bounding box portions of the image data that include each of the food products; and assigning the quality level score to the food product based on the extent to which the food product has yellowed, wherein a score equal to or greater than a first threshold level but less than a second threshold level indicates (i) good quality and (ii) no yellowing, and a score equal to or less than a third threshold level but greater than the second threshold level indicates (i) poor quality and (ii) maximum yellowing of the food product; The method according to any one of claims 1 to 5, comprising:

13. 13. The method of claim 12, wherein applying a lime yellowing trained model further comprises applying a Gaussian mixture model and a support vector regressor to the bounding box portions of the image data that include each of the food items.

14. 12. The method of claim 11 , wherein applying an apple yellowing trained model further comprises applying a Gaussian mixture model and a support vector regressor to the bounding box portions of the image data that include each of the food items.

15. and identifying, by the computing system, for each of the food products, a quality level of the food product. identifying internal quality defects within the food products based on applying an internal quality trained model to the bounding box portions of the image data including each of the food products, the internal quality trained model being trained using image training data of other foods previously annotated as having no internal defects, extensive browning, vascular browning, nuclear desquamation, internal rot, stem rot, and tissue destruction; identifying an overall quality level of the food product based on the internal quality defects, wherein a score of "good" indicates that the food product has no internal quality defects, a score of "acceptable" indicates that the food product has some internal quality defects, and a score of "poor" indicates that the food product has the most internal quality defects; The method according to any one of claims 1 to 5, comprising:

16. and identifying, by the computing system, for each of the food products, a quality level of the food product. identifying spoilage and desiccation of the food products based on applying a spoilage and desiccation trained model to the bounding box portions of the image data that include each of the food products; assigning the quality level score of the food product based on the identified spoilage and dryness, the score being at least one of a binary classification of spoilage and dryness and a numerical value, the numerical value being a continuous value between 0 and 1 indicating the severity of the spoilage and dryness; The method according to any one of claims 1 to 5, comprising:

17. and identifying, by the computing system, for each of the food products, a quality level of the food product. determining the ripeness and remaining shelf life of the food product based on applying a shelf-life trained model to the image data; assigning the quality level score to the food product based on the determined ripeness and remaining shelf life, the quality level score being a numerical value indicating the number of days remaining until the food product is ready to eat; The method according to any one of claims 1 to 5, comprising:

18. and identifying, by the computing system, for each of the food products, a quality level of the food product. determining a grade of the food products based on applying a produce grade trained model to the bounding box portion of the image data that includes each of the food products, the grade being based on USDA specifications, color distribution, shape of the food products, and size of the food products; assigning the quality level score of the food product based on the grade, the score being a string value indicative of an assessment of at least one of the grade, the color distribution, the shape, and the size of the food product; The method according to any one of claims 1 to 5, comprising:

19. 20. The method of claim 18, wherein the string value indicating the assessment of the grade is at least one of "excellent," "top quality," "non-excellent," "good," "acceptable," and "poor."

20. The method of claim 18 , wherein the string value indicative of the assessment of the color distribution is at least one of “color good” and “color bad.”

21. The method of claim 18 , wherein the string value indicative of the assessment of the shape is at least one of “well-formed” and “poorly formed.”

22. The method of claim 18 , wherein the string value indicating the assessment of the size is at least one of “good size” and “bad size.”

23. and identifying, by the computing system, for each of the food products, a quality level of the food product. determining a size of the food items based on applying a size-trained model to the bounding box portion of the image data that includes each of the food items; and assigning the quality level score to the food product based on the size, the score being a numerical value indicating whether the food product fits into a standard bin size for other foods of the same type as the food product; The method according to any one of claims 1 to 5, comprising:

24. and identifying, by the computing system, for each of the food products, a quality level of the food product. identifying shapes of the food items based on applying a shape-trained model to the bounding box portions of the image data that include each of the food items; assigning the quality level score to the food product based on the shape, the score being a numerical value indicative of the bending radius of the food product; The method according to any one of claims 1 to 5, comprising:

25. and identifying, by the computing system, for each of the food products, a quality level of the food product. Identifying (i) the presence, (ii) coverage, and (iii) thickness of a aged coating on the food products based on applying a aged coating trained model to the bounding box portions of the image data that include each of the food products; assigning the quality level scores of the food products based on (i)-(iii), the scores being at least one of binary inferences and numerical values ​​of pixels in the bounding box portion of the image data including each of the food products that indicate the presence, coverage, and thickness of the ripened coating on the food products; The method according to any one of claims 1 to 5, comprising:

26. and identifying, by the computing system, for each of the food products, a quality level of the food product. determining a dry matter content of the food products based on applying a dry matter trained model to the bounding box portions of the image data that include each of the food products; assigning the quality level score of the food product based on the dry matter content, the score being a numerical value indicative of the quantity of the dry matter content of the food product; The method according to any one of claims 1 to 5, comprising:

27. and identifying, by the computing system, for each of the food products, a quality level of the food product. determining the hardness of the food products based on applying a hardness trained model to the bounding box portions of the image data that include each of the food products; assigning the quality level score of the food product based on the firmness, the score being a numeric value; The method according to any one of claims 1 to 5, comprising:

28. and identifying, by the computing system, for each of the food products, a quality level of the food product. identifying product labels on the food products based on applying a product identifier trained model to the bounding box portions of the image data that include each of the food products; decoding at least one of text and a product identifier on the product label using optical character recognition (OCR); assigning the quality level score for the food product based on the decoded product label, the score being a string value including at least one of the decoded text and the product identifier; The method according to any one of claims 1 to 5, comprising:

29. and identifying, by the computing system, for each of the food products, a quality level of the food product. identifying the astringency of the food products based on applying an astringency trained model to the bounding box portions of the image data that include each of the food products; assigning the quality level score to the food product based on the astringency; The method according to any one of claims 1 to 5, comprising:

30. and identifying, by the computing system, for each of the food products, a quality level of the food product. identifying a sourness level of the foods based on applying a sourness trained model to the bounding box portions of the image data that include each of the foods; and assigning the quality level score to the food product based on the sourness level of the food product, the score being a numerical value indicative of the sourness level of the food product; The method according to any one of claims 1 to 5, comprising:

31. and identifying, by the computing system, for each of the food products, a quality level of the food product. calibrating the color of the image data to maintain color consistency in the image data; obtaining a color sample of each food item within the calibrated bounding box of the image data; mapping the color samples into a multi-dimensional color space; identifying a direction of maximum color change across all of the color samples; predicting a location of the maximum color change along the identified direction of the food product, the location of the maximum color change being a one-dimensional quality metric for the food product; and identifying the quality level of the food product as good quality based on determining that the one-dimensional quality metric of the food product is within a threshold quality range; The method according to any one of claims 1 to 5, comprising:

32. The quality level score of the food product is an overall quality metric for the food product, the overall quality metric being: accessing from a data store (i) a rule-based mapping of different ranges of score values ​​corresponding to the identification of the first and second quality characteristics to (ii) enumerated categories of food quality, the enumerated categories of food quality including at least one of salable, unsalable, edible, inedible, good quality, poor quality, and acceptable quality; iteratively determining whether each rule-based mapping is satisfied for each of the first and second quality features; identifying the food quality enumerated category of the food product based on each of the rule-based mappings being satisfied; assigning the identified category to the food product as the overall quality metric for the food product; The method according to any one of claims 1 to 5, wherein the method is identified by performing the following steps.

33. and identifying, by the computing system, for each of the food products, a quality level of the food product. assigning a binary value indicating the presence or absence of wrinkles on the surface of the food product represented in the bounding box portion based on applying a wrinkle analyzer model to the bounding box portion of the image data including each of the food products; (i) summing the quantities of the bounding box portions that are assigned binary values ​​indicating the presence of wrinkles on the surface of the food product represented by the corresponding bounding box portions; and (ii) determining the percentage of wrinkle coverage on the surface of the food product represented by the bounding box portions of the image data based on dividing the sum by the total quantity of the bounding box portions of the image data; assigning the quality level score to the food product based on the identified percentage of wrinkle coverage meeting a threshold wrinkle criterion, the score being a numeric value indicating whether wrinkles are present or absent on the surface of the food product; The method according to any one of claims 1 to 5, comprising:

34. and identifying, by the computing system, for each of the food products, a quality level of the food product. determining a median color value of the foods based on applying a calyx browning trained model to the bounding box portions of the image data that include each of the foods; and assigning the quality level score for the food product based on the median color value of the food product, wherein the quality level score is assigned (i) a string value of "good" based on the median color value being less than a first threshold color range, (ii) a string value of "acceptable" based on the median color value being greater than the first threshold color range and less than a second threshold color range, or (iii) a string value of "poor" based on the median color value being greater than the second threshold color range; The method according to any one of claims 1 to 5, comprising:

35. and identifying, by the computing system, for each of the food products, a quality level of the food product. identifying color values ​​of the foods based on applying a banana stage analyzer trained model to the bounding box portions of the image data that include each of the foods; assigning the quality level score of the food product based on mapping the color values ​​of the food product to threshold color values ​​expected at a plurality of ripeness stages, the quality level score being a string indicating the current ripeness stage of the food product; The method according to any one of claims 1 to 5, comprising:

36. and identifying, by the computing system, for each of the food products, a quality level of the food product. extracting stems of the food products based on applying an object detection model to the bounding box portions of the image data that include each of the food products; determining a median color value of the stems of the food product based on applying a cherry stem color trained model to the extracted stems of the food product; assigning the quality level score of the food product based on the median color value of the stem of the food product, wherein the quality level score is assigned (i) a string value of "good" based on the median color value being within a first threshold color range, (ii) a string value of "acceptable" based on the median color value being within a second threshold color range that does not include the first threshold color range, and (iii) a string value of "poor" based on the median color value being within a third threshold color range that does not include at least the first threshold color range. Allocating and The method according to any one of claims 1 to 5, comprising:

37. The method of any one of claims 1 to 5, wherein the plurality of trained models are trained using at least one of a convolutional neural network (CNN) and a partial least squares (PLS).

38. the image data includes at least one of an RGB image, a hyperspectral image, a multispectral image, a thermal image, a nuclear magnetic resonance (NMR) image, and a magnetic resonance image (MRI); The method according to any one of claims 1 to 5.

39. and identifying, by the computing system, for each of the food products, a quality level of the food product. determining a dry matter content of the food products based on applying a dry matter trained model to the bounding box portions of the image data that include each of the food products; assigning the quality level score to the food product based on the dry matter content, the score being a numerical value indicative of the quantity of the dry matter content of the food product; The method according to any one of claims 1 to 5, comprising:

40. and identifying, by the computing system, for each of the food products, a quality level of the food product. determining sugar content levels of the foods based on applying a Brix-trained model to the bounding box portions of the image data that include each of the foods; and assigning the quality level score of the food product based on the sugar content level, the score being a number indicating the sugar content in Brix units of the food product; The method according to any one of claims 1 to 5, comprising:

41. and identifying, by the computing system, for each of the food products, a quality level of the food product. identifying the nutritional content of target compounds in the foods based on applying a nutrient-trained model to the bounding box portions of the image data that include each of the foods; and assigning the quality level score of the food product based on the nutritional content of the target compounds, the score comprising a list of concentrations of the nutritional content of the target compounds in the food product; The method according to any one of claims 1 to 5, comprising:

42. 1. A system for identifying a quality level of a food product using image data, comprising: one or more imaging devices configured to measure image data of food products of the same food product type; at least one computing system; Equipped with the at least one computing system; receiving image data of the food item from the one or more imaging devices; performing object detection on the image data to identify a bounding box around each of the food items in the image data; identifying a grid structure for the image data based on the bounding boxes around each of the food items in the image data, wherein each bounding box is assigned a grid index in the grid structure, and the grid index is used to identify the food item in a data store; identifying, for each of the food products, a quality level of the food product by applying a plurality of trained models to the bounding box portions of the image data including each of the food products, each of the trained models being trained using image training data of other foods, the image training data being annotated based on previous identification of a first portion of the other food product as having a poor quality feature and a previous identification of a second portion of the other food product as having a good quality feature, the other foods being of the same food type as the food product, and the plurality of trained models including a first trained model trained to identify a first quality feature and a second trained model trained to identify a second quality feature different from the first quality feature; determining, for each of the food products, a quality level score for the food product based on the identified quality level of the food product; storing, for each of the food items, (i) the bounding box portion of the image data that includes the food item, (ii) the grid index, (iii) the identified quality level of the food item, and (iv) the determined quality level score of the food item in the data store; A system configured to:

43. 1. A system for identifying a quality level of a food product using image data, comprising:

1. A photo box having first, second, third, and fourth walls and a ceiling, said photo box comprising: an opening in the first wall configured to receive a shallow box containing food of the same food type; a flap configured to cover the opening and prevent ambient light from entering the photo box; At least one light configured within the photo box to provide consistent lighting to illuminate the shallow box containing the food product; and one or more imaging devices attached to the ceiling of the photo box and configured to capture image data of the food on the shallow box; a photo box, at least one computing system; Equipped with the at least one computing system; receiving image data of the food item from the one or more imaging devices; performing object detection on the image data to identify a bounding box around each of the food items in the image data; identifying a grid structure for the image data based on the bounding boxes around each of the food items in the image data, wherein each bounding box is assigned a grid index in the grid structure, and the grid index is used to identify the food item in a data store; identifying, for each of the food products, a quality level of the food product by applying a plurality of trained models to the bounding box portions of the image data including each of the food products, each of the trained models being trained using image training data of other foods, the image training data being annotated based on previous identification of a first portion of the other food product as having a poor quality feature and a previous identification of a second portion of the other food product as having a good quality feature, the other foods being of the same food type as the food product, and the plurality of trained models including a first trained model trained to identify a first quality feature and a second trained model trained to identify a second quality feature different from the first quality feature; determining, for each of the food products, a quality level score for the food product based on the identified quality level of the food product; storing, for each of the food items, (i) the bounding box portion of the image data that includes the food item, (ii) the grid index, (iii) the identified quality level of the food item, and (iv) the determined quality level score of the food item in the data store; A system configured to:

44. 44. The system of claim 43, wherein the at least one light is an LED light.

45. the photo booth further comprising a camera rig configured to extend across the ceiling of the photo booth; the camera rig includes three tubes connected at a T-connector; a first tube extending from the T-connector to the first wall, and a second tube extending from the T-connector to the second wall; the second wall faces the first wall, and a third tube extends from the T-connector to the third wall; the third tube is perpendicular to the first and second tubes; 45. The system of claim 43 or 44, wherein the one or more imaging devices are mounted to the camera rig along at least one of the three tubes near the T-connector.

46. 45. The system of claim 43 or 44, wherein the one or more imaging devices are at least one of an RGB camera, a hyperspectral imaging device, a thermal imager, an MRI scanning device, and an NMR imaging device.

Images