Ultraviolet light and machine learning based assessment of food item quality

Abstract

The disclosed technology provides for determining an infection in a food item using image data of the food item under ultraviolet (UV) light. A method includes performing object detection on image data to identify a bezel surrounding each food item in the image data; for each food item, a measure of infection presence is determined by applying a model to the bezel, the model being trained using image training data of other food items under UV light, the image training data is annotated based on previously identifying a first portion of the other food item as having an infection feature and identifying a second portion as having a health quality feature; and determining an infection coverage measure for each food item based on determining that the infection presence measure for that food item is indicative of the presence of an infection.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit under 35 U.S.C. §119(e) of U.S. patent application No. 63 / 328,052, filed on April 6, 2022, entitled “ULTRAVIOLETLIGHT AND MACHINE LEARNING-BASED ASSESSMENT OF FOOD ITEM QUALITY,” which is incorporated herein by reference in its entirety. Technical Field

[0003] This document describes apparatus, systems, and methods related to determining the quality of a food item, for example, based on image data of the food item. Background Art

[0004] Food items such as agricultural products, fruits and meats can have different quality measures, which may affect their suitability for consumption and value in the supply chain. Multiple different stakeholders in the entire supply chain are interested in evaluating the quality measures of such food items. As an example, the color of a food item can be a quality indicator that can be used to classify and rate a food item. Different quality measures may also affect consumer purchase decisions.

[0005] Food items with quality measures such as good color attributes may be more valuable than food items with poor color or other poor quality measures. Color and other quality measures may be used to indicate the maturity, hardness, infection, decay, dehydration, taste, sweetness and acidity characteristics of food items. Any of these characteristics may be valuable in the entire supply chain and in the consumer's consumption decision. For example, browning in food items may indicate decay or infection. By identifying the early stage of color browning, supply chain modifications can be made to avoid wasting food items.

[0006] It may be challenging to objectively and quantitatively define high-quality food items based on image data of food items in the visible spectrum. Relevant stakeholders in the supply chain can observe and compare colors or features visible in or on food items. However, the human eye can only distinguish color differences or other feature differences in extreme cases, or can only make distinctions when the color differences or other feature differences are obvious. For example, the human eye may be able to distinguish whether a lime is yellow, green, or brown, or that one lime is more yellow than other limes, but the human eye may not be able to recognize more subtle color changes. As another example, the human eye has challenges in comprehensively arranging and comparing all possible colors that a particular food item can obtain during its life cycle. Subtle changes in color and other features throughout the life cycle of a food item may indicate changes in the quality of the food item. Summary of the invention

[0007] This document generally describes systems, methods and techniques for non-invasively assessing the quality of food items (e.g., vegetables, fruits, meats), for example, based on image data. More specifically, the disclosed technology provides for assessing the quality of food items from image data captured using or under ultraviolet (UV) light conditions. Features of food items that indicate quality (e.g., subtle color changes, areas around bruises or rot or other infections, etc.) may sometimes not be very recognizable in the visible spectrum. UV light can cause potential infections (such as bruises, rot, dehydration, mold, etc.) to fluoresce. Since potential infections are more visible in UV light, image data of food items can be captured under UV light conditions. The image data can include images (e.g., RGB, hyperspectral, multispectral, etc.) and additional metadata. As an illustrative example, the disclosed technology can be used to assess the color of food items to determine the quality of food items, such as the presence of infection, maturity, fitness for consumption, mold, rot, dehydration, etc. The disclosed technology can distinguish between color differences, whether obvious or subtle, to help determine one or more different quality metrics for food items.

[0008] For example, the disclosed technology can identify the presence of infection in citrus fruits (e.g., oranges, limes, lemons, etc.), how much surface area of ​​citrus fruits is covered by infection, the current edibility of citrus fruits based on the presence of infection, and the predicted length of time that citrus fruits remain edible. The disclosed technology can make such determinations for single food items and / or batches of food items of the same food type. At least one model can be trained using machine learning techniques (such as k-means clustering) to determine the quality measures for food items mentioned above. The at least one model can also be trained to identify various other quality measures (such as shelf life, maturity, rot, mold, dehydration, bruises, mildew, etc.) for various other types of food items (such as avocados, apples, berries, mangoes, cucumbers, bell peppers, etc.). As a result, the quality of food items can be evaluated, especially when the human eye may not be able to provide objective and comprehensive quantification or may not be able to distinguish subtle changes in food items (such as bruises, infection, aging, maturity, taste, etc.) or other invisible features of the quality of food items. In addition, different types of infections (e.g., latent infections, wound infections) have different fluorescent signatures. Models trained by machine learning techniques (e.g., k-means clustering) can be used to identify each of these unique signatures, thereby identifying such types of infections with high accuracy and high throughput. Based on the detected infections, the disclosed technology can also determine and predict the shelf life of food items and / or the length of edible time for food items, so that appropriate supply chain modifications can be made to eliminate or reduce waste of food items.

[0009] The disclosed technology can provide for generating different metrics that can be used to identify different features of the food item that indicate the quality of the food item. A quality metric can be defined for each food item. A quality metric can also be defined for each type of food item. For example, apples can have different color metrics and corresponding (multiple) machine learning trained models to identify infection or other quality metrics / features compared to limes, avocados, oranges, lemons, limes, and other fruits and agricultural products.

[0010] The disclosed technology can also be used to determine the existence of infection in food items. Based on the determined infection existence, the supply chain can be modified in the early stage. For example, once the food item enters the storage facility, the quality assessment described in the present disclosure can be implemented. The quality of the food item can be evaluated at this time point, which can be an important indicator about how long the food item is edible or consumable (even if there are some infections in the food item). If the quality of the food item is identified as poor (for example, the number of infections that the food item has exceeds a certain threshold health level), the supply chain can be modified to immediately ship the food item to the consumer at the nearest geographical location (for example, if the disclosed technology determines that the food item is still consumable for a certain amount of time), or the food item is shipped to a food processing plant. If the quality of the food item is identified as good (for example, the food item is not infected or the number of infections that it has is lower than the threshold health level), the supply chain can be modified to store the food item for a period of time, or the food item is transported to the consumer at a farther geographical location. Based on the preferences of the relevant stakeholders in the entire supply chain life cycle and the quality measurement of the determined agricultural product, one or more other supply chain modifications can be determined.

[0011] One or more embodiments described herein may include a method for determining an infection in a food item using image data, the method comprising: receiving, by a computing system, image data of a food item under ultraviolet (UV) light from an imaging device; performing, by the computing system, object detection on the image data to identify a bounding box surrounding each food item in the image data; determining, by the computing system, a grid structure for the image data based on the bounding box surrounding each food item in the image data, each bounding box being assigned a grid index in the grid structure, the grid index being used to identify the food item in a data repository; determining, by the computing system, for each food item, an infection presence measure for the food item by applying a model to a bounding box portion of the image data, the model being determined using The method comprises the steps of: training the image training data of other food items under UV light, wherein the image training data is annotated based on previously identifying a first portion of the other food items as having infection characteristics and identifying a second portion of the other food items as having health quality characteristics, wherein the other food items belong to the same food type as the food item; determining, by a computing system, an infection coverage measure for each food item based on determining that an infection presence measure for the food item indicates the presence of infection; and returning, by the computing system, for each food item: (i) a bounding box portion of the image data containing the food item, (ii) a grid index, (iii) an infection presence measure for the food item, and (iv) an infection coverage measure for the food item.

[0012] In some implementations, the embodiments described herein may optionally include one or more of the following features. For example, the infection coverage measure for a food item may be the percentage of the surface of the food item including features indicating infection. The food item in the image data may include at least one of citron, mandarin orange, pomelo, orange, grapefruit, lemon, lime and tangerine. The infection presence measure may be a string value indicating a healthy food item or an infected food item.

[0013] The method may also include: identifying, by a computing system, a group of food items in the image data as having respective infection presence measures that satisfy a threshold infection level, the threshold infection level indicating that the food items are infected; determining, by a computing system, an edibility measure for the group of food items by applying an edibility model to the image data, the edibility model being trained using training time series image data of infected food items, the training time series image data being annotated by prior identification of infection surface coverage associated with an edible length of time for the infected food items, the infected food items belonging to the same food type as the food items; and returning, by the computing system, an edibility measure for the group of food items. The edibility measure may indicate whether the group of food items is edible. The computing system determining the edibility measure for the group of food items may include predicting an edible length of time for the group of food items. The edibility measure may indicate an edible length of time for the group of food items.

[0014] As another example, a computing system trains a model using a process that includes the following steps: receiving image training data of other food items; performing object detection on the image training data to identify a training bounding box around the food item; identifying healthy features, infected features, and background features of the food item in the bounding box; mapping the identified features into a multi-color space; and training the model using a clustering algorithm to determine an infection presence measure for the food item based on the identified features mapped into the multi-color space. The clustering algorithm can be k-means. In some implementations, the method can include: the computing system sends an infection presence measure for each food item in the image data to a user computing device for display in a graphical user interface (GUI).

[0015] The method may also include: the computing system retrieves an infection presence measure for each food item from a data repository; the computing system identifies supply chain information for the food item including an existing supply chain schedule and a destination for the food item; the computing system determines whether to modify the supply chain information for the food item based on the infection presence measure; in response to determining to modify the supply chain information, the computing system generates modified supply chain information based on the infection presence measure, the modified supply chain information including one or more of the modified supply chain schedule and the modified destination for the food item; and the computing system sends the modified supply chain information to one or more supply chain participants to implement the modified supply chain information. The computing system determines whether to modify the supply chain information for the food item based on the infection presence measure, including determining that the infection presence measure meets a threshold infection level, the threshold infection level indicating that the food item is infected rather than healthy. The modified supply chain information may include instructions that, when executed by one or more supply chain participants, cause the food item to be moved for outbound shipment to an end consumer at a location that is geographically closest to the food item. The modified supply chain information may include instructions that, when executed by one or more supply chain participants, result in at least one of: (i) the food item being moved for shipment to a food processing plant, or (ii) controlled atmospheric conditions surrounding the food item being changed.

[0016] As another example, the image data may include at least one of an RGB image, a hyperspectral image, a multispectral image, a nuclear magnetic resonance (NMR) image, and a magnetic resonance image (MRI). In addition, determining, by the computing system, for each food item, a measure of the presence of infection for the food item includes: determining that the food item is infected based on an infection coverage of the food item exceeding a threshold infection coverage. The method may also include: determining that the food item is healthy based on an infection coverage of the food item being less than a threshold infection coverage.

[0017] In some implementations, determining an infection coverage measure for each food item by a computing system based on determining that an infection presence measure for the food item indicates the presence of an infection may include: summing pixels in the image data that represent healthy characteristics of the food item and unhealthy characteristics of the food item; dividing the number of pixels in the image data that represent unhealthy characteristics of the food item by the summed pixels to generate an infection coverage value; and multiplying the infection coverage value by a predetermined factor to generate a percentage value as the infection coverage measure.

[0018] One or more embodiments described herein may include a system for determining the presence of an infection in a food item using image data, the system comprising: at least one light source that can illuminate food items of the same food type, the at least one light source emitting ultraviolet (UV) light; one or more imaging devices that can capture image data for the food items when the food items are illuminated by the at least one light source; and at least one computing system in communication with the one or more imaging devices, the at least one computing system being configured to: receive image data of the food items under UV light from the one or more imaging devices; perform object detection on the image data to identify a bounding box surrounding each food item in the image data; determine a grid structure for the image data based on the bounding box surrounding each food item in the image data, wherein each bounding box is assigned a grid index in the grid structure, and the grid index is used to store the bounding box in the data Identifying food items in a repository; determining, for each food item, an infection presence measure for the food item by applying a model to a bounding box portion of the image data, the model being trained using image training data of other food items under UV light, the image training data being annotated based on previously identifying a first portion of the other food item as having infection characteristics and a second portion of the other food item as having health quality characteristics, wherein the other food items are of the same food type as the food item; determining an infection coverage measure for each food item based on determining that the infection presence measure for the food item indicates the presence of an infection; and returning, for each food item: (i) a bounding box portion of the image data containing the food item, (ii) a grid index, (iii) an infection presence measure for the food item, and (iv) an infection coverage measure for the food item.

[0019] The system may optionally include one or more of the following features. For example, the at least one light source may be a UV LED lamp. The at least one light source may be a black light lamp. The at least one light source may be a UV floodlight. The at least one light source may emit UV light having a wavelength in the range of approximately 300nm-400nm. The infection coverage measure for the food item may be a percentage of the surface of the food item that includes features indicating infection. The food item in the image data may include at least one of citron, mandarin orange, pomelo, orange, grapefruit, lemon, lime, and tangerine.

[0020] In some implementations, the at least one computing system may also: identify a group of food items in the image data as having respective infection presence measures that satisfy a threshold infection level, the threshold infection level indicating that the food items are infected; determine an edibility measure for the group of food items by applying an edibility model to the image data, the edibility model trained using training time series image data of infected food items, the training time series image data annotated by prior identification of infected surface coverage associated with an edible length of time of the infected food items, wherein the infected food items are of the same food type as the food item; and return the edibility measure for the group of food items. The image data may be the time series image data for the group of food items.

[0021] As another example, the at least one computing system may determine an infection coverage measure for each food item based on determining that an infection presence measure for the food item indicates the presence of an infection based on the following steps: summing pixels in the image data that represent healthy characteristics of the food item and unhealthy characteristics of the food item; dividing the number of pixels in the image data that represent unhealthy characteristics of the food item by the summed pixels to generate an infection coverage value; and multiplying the infection coverage value by a predetermined factor to generate a percentage value as an infection coverage measure.

[0022] One or more embodiments described herein may include a method for determining the presence of infection in a food item using image data, the method comprising: receiving, by a computing system, image data of a food item under UV light from an imaging device; determining, by the computing system, a measure of the presence of infection for the food item by applying a model to the image data, the model being trained using image training data of other food items under UV light, the image training data being annotated based on previously identifying a first portion of the other food item as having an infection characteristic and a second portion of the other food item as having a healthy quality characteristic, wherein the other food item is of the same food type as the food item; and returning, by the computing system, a measure of the presence of infection for the food item in the image data.

[0023] The method may optionally include one or more of the following features. For example, the method may include: determining, by a computing system, an infection coverage measure for a food item based on the infection presence measure indicating the presence of an infection; and returning, by a computing system, an infection coverage measure for the food item in the image data. In some implementations, returning, by a computing system, an infection presence measure for the food item in the image data may include: sending the infection presence measure and the infection coverage measure to a user device for presentation in a GUI display at the user device.

[0024] The devices, systems, and techniques described herein can provide one or more of the following advantages. For example, the disclosed techniques can provide accurate assessments of quality features of food items that may not be easily detected in the visible spectrum through image data or with the human eye. Thus, image data of food items can be captured under UV light conditions. UV light can cause potential infections (such as bruises, rot, dehydration, mildew, etc.) to fluoresce. Since potential infections are more visible in UV light, image data captured under such conditions can be used to more accurately assess food item quality than under other lighting conditions.

[0025] In addition, by using a machine learning trained model, the quality of food items can be more accurately determined from subtle differences that appear in image data captured under UV light conditions. The human eye may be prone to error when trying to observe subtle changes in the appearance of food items, and may not be able to detect invisible features of food items such as latent or traumatic infections. For example, the human eye may not notice a slight discoloration on the portion around the mold spot on an orange, which may indicate early signs of rot or other poor quality features, such as a shortened length of time available for consumption. In addition, workers will have to be trained to visually measure the color and quality of food items in a quantitative manner. This can be a tedious and time-consuming process, and human bias may occur. The disclosed technology provides for automatic and accurate detection of different quality features in food items from (multiple) high-quality labeled and / or unlabeled data sets. The disclosed technology provides for analyzing the quality of food items beyond the visible spectrum and color, all of which may be challenging or impossible for human eye observation and analysis. Therefore, the disclosed technology can provide a deeper analysis, thereby improving efficiency and reducing human errors that may occur from observing visible features of food items.

[0026] As another example, the disclosed technology can be used to make appropriate supply chain modifications early enough in the supply chain life cycle to reduce waste based on food items. The quality of food items can be evaluated at any point in the entire supply chain. For example, the quality can be evaluated before the food items are shipped from the farm to the storage facility. The quality can also be evaluated once the food items arrive at the storage facility. In some implementations, the quality can even be evaluated once it is on the shelf in a grocery store and available to consumers. When the quality of food items is evaluated early in the supply chain life cycle, such food items can be more appropriately classified based on the identified and / or expected quality of the food items. For example, compared with food items identified as poor quality (e.g., with a certain number of potential or traumatic infections), food items identified as good quality (e.g., without potential or traumatic infections) when entering the storage facility can be stored in the facility for a longer time. Improved decisions can also be made about how and when food items are appropriately processed. For example, if one or more food items are determined to not meet a threshold quality level (e.g., have a threshold number of infections present), the disclosed technology can determine that the appropriate supply chain modification is to initiate the application of an antimicrobial treatment to the food item. The disclosed technology can also be used to determine consumption and edibility time frames, which can affect when food items are delivered to grocery stores, how long food items are kept in storage, and whether food items should be delivered to food processing plants instead of grocery stores.

[0027] As described throughout, the disclosed technology can generate a robust quality assessment for food items. Machine learning techniques can be used with high-quality labeled training data sets to generate and train different models to identify and score different quality features associated with different food items. Such a robust quality assessment can be useful for more accurately monitoring food item quality and modifying the supply chain accordingly to reduce or eliminate food item-based waste.

[0028] As another example, the disclosed technology provides for evaluating the quality of food items in a non-destructive manner. Since the model is trained to analyze the quality characteristics of food items from image data under UV light conditions, humans do not have to implement destructive techniques such as punching or squeezing on the epidermis, surface or flesh of the food items to determine their quality. As a result, the food items delivered to the end consumers can be accurately tested and quality evaluated without actually reducing the quality of the food items. Therefore, the model is trained to extract quality features from the image data of the food items, rather than requiring humans to destroy or otherwise alter the food items before they are delivered to the consumers. When the disclosed technology is used to evaluate the quality of food items, higher quality food items can be delivered to consumers, and food items can be saved from waste.

[0029] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features and advantages will become apparent from the following description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1A A conceptual diagram of determining the quality of a food item based on image data under ultraviolet (UV) light.

[0031] Figure 1B is a conceptual diagram of a model generated to determine the quality of a food item.

[0032] Figure 1C is an illustration of an exemplary system for assessing the quality of one or more food items based on image data.

[0033] Figure 2A -B is a flow chart of a process for determining the quality of a food item based on image data.

[0034] Figure 3A -B is a flow chart of a process for determining the quality of a food item based on image data.

[0035] Figure 4 is a flow chart of a process for determining the length of time a food item is edible based on its determined quality.

[0036] Figure 5A Depicted is an exemplary infection detection analysis using image data of food items under UV lighting.

[0037] Figure 5B Depicted is an exemplary infection detection analysis using image data of batches of food items under UV illumination.

[0038] Figure 6is an exemplary system diagram of components used to determine the quality of a food item based on the techniques described herein.

[0039] Figure 7 is a schematic diagram illustrating examples of computing devices and mobile computing devices.

[0040] Like reference numbers in the various drawings represent like elements. DETAILED DESCRIPTION

[0041] This document generally relates to a technique for assessing the quality of one or more food items from image data captured under UV light conditions. UV light conditions can reveal damage, infection, or other poor / unhealthy quality features in food items that may not be visible or obvious under visible light conditions. UV-A-induced fluorescence (such as light with a wavelength in the range of 300nm to 400nm) can cause unhealthy quality features in food items to fluoresce, thereby making accurate quality assessment of such food items more visible. The disclosed technique can provide image data of one or more food items obtained under UV light conditions, and determine a quality measure for each food item represented in the image data based on an assessment of the image data. A machine learning technique can be used to train a model to process the image data and determine the quality measure. The quality measure can include the presence of an infection of the food item. Based on the determined presence of an infection, the disclosed technique can also provide for determining the shelf life of the food item, the length of time the food item can be eaten (even if the food item is infected), and other quality measures. Based on the food item and / or food item type and using the image data of the food item under UV light conditions, one or more additional or other quality measures can also be modeled and determined. Additionally, the disclosed techniques may provide for modifying one or more supply chain operations based on a quality assessment of a food item in an attempt to mitigate losses that may be incurred due to food items whose quality levels do not meet a particular threshold.

[0042] With reference to the accompanying drawings, Figure 1A is a conceptual diagram of determining the quality of a food item based on image data under ultraviolet (UV) light. Computer system 150, imaging device 160, and user device 170 may communicate (e.g., wired and / or wirelessly) via network(s) 180. As described throughout this disclosure (e.g., with reference to Figure 1C , 6), the computer system 150 can be configured to assess the quality of the imaged food items (such as produce). The imaging device 160 can include an image sensor 106. The imaging device 160 can be a camera that captures still images, time series, and / or video data. The imaging device 160 can be an RGB camera that captures light primarily in the visible range. Any other camera designed to capture visible light can be used, including but not limited to cameras of mobile devices (e.g., smartphone cameras, tablet cameras, laptop cameras, computer cameras, etc.) and professional or non-professional cameras (e.g., DSLR cameras). UV fluorescent imaging can be used for the disclosed technology, where the peak of the emitted light is near a wavelength of approximately 520nm. The imaging device 160 can be used in an automatic mode to capture images of food items 102A-N under UV light conditions. The imaging device 160 can also be manually set for low light settings. For example, the shutter speed of the imaging device 160 can be 1 / 8, its aperture can be set to F / 3.5, and its ISO can be set to 6400. Under the UV light conditions described throughout this disclosure, one or more other camera settings may be used to capture images of food items 102A-N.

[0043] The at least one light source 105 may also be configured to be in communication with the imaging device 160 (eg, see Figure 1C ). In some implementations, light source 105 can be separate from imaging device 160. Light source 105 can be positioned near imaging device 160 to provide uniform illumination over the area where imaging device 160 will capture images of food items 102A-N.

[0044] The light source 105 may be a UV light source. For example, the light source 105 may emit light having a wavelength in the range of 300nm to 400nm. As other non-limiting examples, the light source 105 may be a UV-B light source that emits light having a wavelength in the range of approximately 200nm to 350nm (e.g., 280nm to 315nm). As another example, the light source 105 may be a UV-A light source that emits light having a wavelength in the range of approximately 300nm to 420nm (e.g., 315nm to 400nm). The light source 105 may also emit light having a wavelength in the range of one or more other non-limiting examples, including but not limited to 500nm to 555nm, 520nm to 570nm, and 420nm to 620nm. In some implementations, for example, the light source 105 may emit light having a wavelength of 365nm. In some implementations, the light source 105 may be a UV LED lamp, a UV LED black light, a high-performance LED bulb, and / or a UV floodlight. For example, light source 105 can be an LED UV lamp reaching 50W. Light source 105 can also be an LED UV lamp as low as 4W, which can be used in combination with an imaging device set to manual or other specific camera settings to pick up the fluorescence signal from food items 102A-N. Multiple light sources 105 can also be positioned around imaging device 160 to provide uniform lighting conditions. As described herein, UV light interacts with the surface of food items 102A-N and triggers fluorescence (in the visible light range). The output of imaging device 160 can be a conventional RGB image presented by food items 102A-N under the irradiation from UV light (which makes some defects such as infection become significant).

[0045] In some implementations, one or more narrow band pass filters can be used between UV light source 105 and irradiated food items 102A-N. Such filters can help reduce visible light emission and use (multiple) specific excitation wavelengths as targets. These filters can, for example, use 285nm, 330nm, 365nm, 395nm or 400nm and any other wavelength of interest as targets. (Multiple) narrow band pass filters can also be used between irradiated food items 102A-N and imaging device 160, so that only the light emitted by the fluorophore (e.g., fluorescent chemical compound emitted by food items 102A-N under UV light conditions) of food items 102A-N is captured. These filters can also use the wavelength between 400nm to 780nm as targets, and can particularly use 520nm as targets, to help accurately capture the image of food items 102A-N, because the infection in food items 102A-N emits fluorescence under UV light conditions. Imaging device 160 and light source 105 can be positioned at various positions along the supply chain of food items 102A-N. For example, imaging device 160 and light source 105 can be placed in a warehouse or other storage facilities along conveyor belt 104 system, and the conveyor belt 104 system transports food items 102A-N to various positions in the warehouse. As described in the application number 63 / 295,172 of the title "MACHINE LEARNING-BASED ASSESSMENT OF FOOD ITEM QUALITY (evaluation based on machine learning of food item quality)" incorporated in full by reference herein, imaging device 160 and light source 105 can also be positioned in a closed environment such as a photo box (photo box) etc. receiving food items 102A-N (e.g., batches, trays or platforms of food items 102A-N), so that uniform illumination and environmental conditions allow to capture accurate and uniform images. The photo box, for example, can be a closed unit that eliminates other light sources (such as ambient light) from outside the light source 105. Imaging performed using the photo box can be conducive to determining the risk of deterioration within a given time range for the batch of food items 102A-N placed therein. In addition, two or more images of food items 102A-N inside the light box may be captured to capture the entire surface of food items 102A-N. As a result, the disclosed techniques may be applied to multiple images of food items 102A-N to determine the overall presence of infection on the total surface of each food item 102A-N, rather than just one side of food items 102A-N.

[0046] In some implementations, the part with imaging device 160 and light source 105 of conveyor belt 104 can be covered with material or some type of enclosure (such as light box), to avoid visible light penetration.For example, two groups of rubber-style curtains or other materials can be positioned at the entry and exit points of the enclosure around the described part of conveyor belt 104.The roller of conveyor belt 104 can be activated when food items 102A-N are in the enclosure inside, so that food items 102A-N wait until they are moved out of the enclosure by moving conveyor belt 104 and have completed a complete rotation.By rotating food items 102A-N while being in the enclosure inside, the whole surface of food items 102A-N can be imaged.Disclosed technology can be used to such an image to determine that the infection on the whole surface of each food item 102A-N exists.

[0047] By using an enclosure, images of food items 102A-N can be captured in the covered portion of conveyor belt 104 without visible light penetrating. The interior surfaces of the enclosure or light box described above can also be coated with a material to limit reflections wherever the image is captured by imaging device 160, since reflections can interfere with the fluorescent signal in the image captured by imaging device 160. The material used to build / construct the enclosure or light box can be selected so that it does not fluoresce at the excitation wavelength(s) of the light emitted by light source 105.

[0048] In some implementations, imaging device 160 may be a handheld device, such as a mobile phone or tablet device, that can be used by a human user to capture images of food items 102A-N. Light source 105 may be fixed to imaging device 160 so that when the handheld device is moved across / passed by food items 102A-N, light source 105 may illuminate food items 102A-N. Figure 1A As depicted in , the imaging device 160 and light source 105 can be positioned above the conveyor belt 104 in the storage facility.

[0049] Imaging device 160 may be configured to continuously capture image data of food items 102A-N as they are moved along conveyor belt 104 to one or more storage locations or other destinations within a storage facility (Block A). Images may be captured under UV light provided by light source 105.

[0050] Food items 102A-N may include, but are not limited to, fruits, citrus fruits, vegetables, and meats. For example, food items 102A-N may include fruits of the genus Citrus, including but not limited to citron (C. medica), mandarin (C. reticulata), grapefruit (C. maxima), and hybrids thereof, such as oranges, grapefruits, lemons, limes, and tangerines. As another example, food items 102A-N may include other types of fruits and agricultural products, including but not limited to apples, mangoes, avocados, cucumbers, and the like.

[0051] Here, food items 102A-N from shippers can be received at storage facilities and loaded onto conveyor belt 104. Food items 102A-N can be in boxes, containers, on trays, platforms, and / or placed directly on conveyor belt 104. In some implementations, food items 102A-N can be static when images are captured by imaging device 160, or are not moved in storage facilities on conveyor belt 104. For example, as described in "MACHINE LEARNING-BASED ASSESSMENT OF FOOD ITEM QUALITY (machine learning-based assessment of food item quality)" application, some of which food items 102A-N can be randomly sampled and placed inside light boxes. Cameras positioned in light boxes can capture image data of food items 102A-N under uniform UV illumination conditions, which can be analyzed by computer system 150 to obtain an estimate of the quality distribution of batches of food items 102A-N.

[0052] The imaging device 160 can send the image data to the computer system 150 (box B). The computer system 150 can then apply one or more models to the image data to identify the features of the food items 102A-N (box C). The features can indicate the quality of the food items 102A-N, such as whether the food items 102A-N are healthy or infected. The features identified using the model can include the color or texture on the surface, epidermis or cortex of the food items 102A-N, which is related to the health and infected quality measurement of the food items for this particular type. The fluorescent color features of the food items 102A-N in the image data can be trained (multiple) models to identify and classify, wherein different quality features indicate the health status of the food items 102A-N (e.g., browning, rotting, deterioration, dehydration, mildew, bruises, potential infection, trauma infection, etc.).

[0053] Therefore, in box C, computer system 150 can apply (multiple) models to image data to identify different color features of food items 102A-N indicating the health status of food items 102A-N. Machine learning techniques can be used to train (multiple) models to identify the features of different food items. Computer system 150 can select one or more models to be applied in box C based on the type of food items 102A-N identified in image data. In addition, as described throughout the present disclosure, computer system 150 can apply models to each food item 102A-N identified in image data to determine the quality measure (e.g., healthy, infected, rotten, moldy, etc.) of each food item. Computer system 150 can also apply models to food items 102A-N of a batch or platform captured in image data to determine the overall quality measure of food items 102A-N for the batch or platform.

[0054] Computer system 150 can determine the quality measure for food items 102A-N based on the identified features in box D. The quality measure can be infection presence. Therefore, computer system 150 can determine whether there is infection in single food items and / or batches of food items 102A-N in image data based on the output of (multiple) models from box C. For example, if computer system 150 determines that food items 102A-N are infected, computer system 150 can also use image data and / or the output of (multiple) models from box C to determine the percentage of each food item being infected and covered. Computer system 150 can also determine the current edibility (such as shelf life, consumption rate, freshness, maturity, etc.) of food items 102A-N based on whether there is infection. In addition, computer system 150 can determine / predict that food items 102A-N may remain edible or available for consumption. As described herein, such determination / prediction can contribute to determining one or more supply chain modifications for food items 102A-N.

[0055] The output from the (multiple) models in box C can be a Boolean value (e.g., yes / no, true / false, 0 / 1, etc.) indicating whether there is an infection. The output can also include a numerical value on a predetermined scale, wherein the numerical value indicates the possibility of infection in the food items 102A-N (e.g., on a scale of 0 to 100, 0 can indicate that the food items 102A-N are healthy and there is no infection, and 100 can indicate that the infection exists / is present in large quantities on the food items 102A-N). In some implementations, the (multiple) model outputs can include instructions about whether there is an infection in each food item 102A-N. In box D, the computer system 150 can then aggregate the various instructions to determine whether there is an infection for the entire batch of food items 102A-N.

[0056] In block D, the computer system 150 may normalize the output from the model(s) to determine a quality measure for the food items 102A-N. For example, the computer system 150 may adjust the output to an infection classification scale indicating the deterioration or infection of the food items 102A-N. The scale may be a numerical scale, including but not limited to 0 to 3.5, 0 to 4, 1 to 3, 1 to 5, 1 to 4, etc. A lower number on the scale may indicate that the food items 102A-N are healthy, with only a few infections present (e.g., infection present below a threshold infection level, which may be specific to the type of food item or other characteristics of the growth / source conditions of the food item and / or the food item), or no infection present. A higher number on the scale may indicate that the food items 102A-N are unhealthy / infected (e.g., a higher number of deteriorations in a specific food item, a batch of food items 102A-N, etc.) and that there is an infection present (e.g., infection present above a threshold infection level). By using the disclosed technology, the computer system 150 can identify different types of infections, such as blue and green mold caused by Penicillium, which can often be manifested as clear circular signals emitted from the wound position in the food project. The disclosed technology can also be used to detect potential types of infections, such as Chromatids, Colletotrichum and Alternaria. Each of these infections can have different types of signals (such as drips from the stem end or buds of the food project, etc.), and (multiple) models can be trained to detect, identify and / or quantify them to determine the quality measurement for food items 102A-N. In addition, as described herein, (multiple) models can be trained to detect various types of infections in various types of food projects, including but not limited to citrus fruits, agricultural products, vegetables such as cucumbers and peppers, avocados, apples, berries and / or other types of food projects that may give rise to such pathogens and infections.

[0057] The computer system 150 may send the quality measurements for the food items 102A-N to the user device 170 (Block E). For example, the computer system 150 may send each quality measurement for each food item 102A-N in the image data. The computer system 150 may send only some quality measurements for some food items 102A-N. The computer system 150 may also send an overall quality measurement for a batch or platform of food items 102A-N in Block E.

[0058] In some implementations, the computer system 150 can also send the quality measure to a database for storage. The quality measure can be stored with other historical measurements and additional metadata associated with batches of food items 102A-N and / or the individual food items in the batches. This information stored can be used in a feedback loop to continuously improve and train the model of the machine learning training that is used to implement the technology described herein. For example, by using the quality measure previously determined, one or more higher-level models can be improved and / or trained to identify the quality measure that depends on seasonality, variety, size, country of origin and other factors.

[0059] User device 170 can output quality measurement in box F. User device 170 can be a mobile device, smart phone, tablet device, laptop computer or other computer that can be used by the relevant stakeholders in the supply chain. In some implementations, the stakeholders can be supply chain participants. The stakeholders can view the quality measurement for each food project 102A-N (and / or batch) to understand or analyze the return on investment (ROI) for food project 102A-N. The measurement exported can also be used by the stakeholders to monitor the quality of food project 102A-N over time, and optionally make one or more supply chain modifications based on the current and / or expected quality (such as edibility, consumption rate, maturity, hardness, infection, etc.) of food project 102A-N.

[0060] Optionally, the user device 170 may determine one or more supply chain modifications based on the output quality metrics for the food items 102A-N (Block G). The user device 170 may automatically determine or otherwise recommend supply chain modifications for one or more food items 102A-N based on their corresponding quality scores. In some implementations, the computer system 150 may determine or otherwise recommend supply chain modifications and send these recommendations to the user device 170. Stakeholders at the user device 170 may optionally implement, modify, or reject any recommended supply chain modifications. In some implementations, stakeholders may review the output quality metrics and determine supply chain modifications to be implemented.

[0061] Although described with reference to quality measures for determining the presence of infection in food items Figure 1A The disclosed techniques may also be used to determine whether a food item is infected by a potentially infectious pathogen. In addition, although Block AG is described with reference to determining the presence of an infection in a batch of food items, Block AG may also be implemented to determine the presence of an infection in a single food item.

[0062] Figure 1B is a conceptual diagram of a process 195 for generating a model for determining the quality of a food item.As described throughout this disclosure, models may be generated for different food items, different types of food items, and different features associated with particular food items.

[0063] The computer system 150 may receive food item image data 190 in block A. The image data 190 may include digital RGB images, hyperspectral images, and / or multispectral images depicting specific food items, types of food items, different food items, different types of food items, single food items, and / or batches of food items. As described throughout the present disclosure, the food item image data 190 may include images of food items captured under UV light conditions.

[0064] The image data 190 may also include images, tables and / or other data of specific food items with certain specific features to be modeled (such as decay and dehydration), as well as images of the same type of food items without the specific features to be modeled. For example, the food item image data 190 may include a table stored in a data repository containing features (such as decay, dehydration, probability of determining shelf life, etc.) extracted, labeled and / or annotated (such as automatically performed by a computer system such as computer system 150, and / or manually performed by relevant stakeholders or other users) from the image of the food item. In some implementations, the food item image data 190 may include images of the exterior of the food item and / or the interior of the food item. In some implementations, the food item image data 190 may include images of specific food items at different maturity stages and between each maturity stage. The image data 190 may also include images of different stages of infection development (such as browning, decay, mildew, dehydration, bruises, etc.) of specific food items within a predetermined time period.

[0065] The food project image data 190 can be the strong total set of the training data of a plurality of different characteristics that show that may exist and / or generate for a specific food project in the whole life cycle of this food project.Image data 190 can also be the total set of the image of the same food project from different angles, thereby can use technology described herein to analyze the whole food project comprehensively.In addition, in some implementations, image data 190 can comprise the label for the feature, condition and / or quality of food project.In other implementations, can use image data 190 that do not comprise label to learn this type of feature, condition and / or quality of food project.

[0066] As previously mentioned, computer system 150 receives image data 190 (Block A). Image data 190 may be received from one or more imaging devices, such as imaging device 160 described herein. Image data 190 may also be retrieved from a data repository.

[0067] The computer system 150 can identify and cluster the features of the food items from the image data in box B. In some implementations, the computer system 150 can implement object detection techniques to identify each food item in the image data 190. The computer system 150 can then randomly select one of the identified food items for the purpose of training. Therefore, the computer system 150 can use one of the food items in the image data 190 to generate and train the (multiple) models described herein. In some implementations, the computer system 150 can use more than one food item in the image data 190 to generate and train the (multiple) models described herein.

[0068] In block B, the computer system 150 may identify features that indicate the quality of the food item. For example, the computer system 150 may identify from the RGB image data the color of the food item that is associated with and / or corresponds to rot, mold, different types of textures, bruises, or other types of infestations. The identified features may be labeled as further described throughout this disclosure. Object detection techniques may be applied to the image data to detect the food item, identify the food item, and then additional extraction steps may be applied to single out one or more specific features of the food item.

[0069] The computer system 150 can also cluster the food item features based on its color data in box B. The computer system 150 can cluster the features based on its color using k-means clustering techniques. The computer system 150 can also use other types of clustering techniques. The computer system 150 can generate three clusters. One cluster can represent the background in the image data 190 (e.g., the platform, tray, container, conveyor belt or other surface on which the food item is parked). Another cluster can represent the healthy features of the surface on the food item in the image data 190 (e.g., the epidermis or cortex of the food item of the color that does not include the color that indicates infection, decay, mildew, dehydration, browning, bruises, etc.). Another cluster can represent the unhealthy or infected features on the surface of the food item in the image data 190 (e.g., the epidermis or cortex of the food item of the color that includes the color that indicates infection, decay, mildew, dehydration, browning, bruises, etc.). In some implementations, the computer system 150 can generate fewer or more clusters. For example, the computer system 150 may generate one or more additional clusters to represent one or more other characteristics of the food items that will determine their food item quality.

[0070] In block C, the computer system 150 may map the clusters of food item features in a multi-color space. The multi-color space may be an RGB space. One or more other color spaces may be used for mapping purposes. In block C, the computer system 150 may map the clusters of infected features, the clusters of healthy features, and the clusters of background features to different colors in the multi-color space.

[0071] The computer system 150 can then generate (multiple) machine learning models using the mapped clusters (box D). In some implementations, the model can be generated and / or trained by one or more other computing systems, computers, device networks, and / or cloud-based services. The model can be trained, for example, by a remote computer system, stored in a data repository, and accessible and executable by the computer system 150. The model can be generated using machine learning techniques, including but not limited to k-means clustering or other clustering techniques. Convolutional neural networks (CNNs) and other machine learning techniques can also be used for training purposes. The computer system 150 can generate a model for each identified and labeled feature. Each model can also be trained to score the quality of the food item based on the identified features. As an example, a model can be trained to determine the presence of infection in oranges and score the number of infections present in oranges.

[0072] For example, the computer system 150 may retrieve image data of a platform containing oranges. The computer system 150 may process the image data in RGB space. Using object detection techniques, the computer system 150 may identify each orange in the image data and generate a bounding box around each orange. The computer system 150 may select (e.g., randomly) one of the bounding boxes to analyze the orange therein in the RGB space. The computer system 150 may train a model using a k-means clustering technique to identify and cluster color features that indicate background, healthy parts of oranges, and unhealthy or infected parts of oranges. The clusters may be visualized in RGB space to accurately identify the center of infection, the spread of infection on the surface of the orange, the healthy part of the surface of the orange (uninfected part), and the background surrounding the orange. The model may also be trained to output an indication of whether there is infection in the orange. The indication can be a Boolean value (e.g., true / false, yes / no, unhealthy / healthy, infected / not infected, etc.), a string value (e.g., "infected food item"), and / or a numeric value (e.g., on a scale of 0 to 5, with 0 being healthy / not infected and 5 being the unhealthiest / most infected).

[0073] The generated model may then be output by the computer system 150 (Block E). During runtime, the generated model may be applied to the image data to identify features indicative of quality in the imaged food item. Outputting the generated model may include presenting the model to a user at a user device. The user may then select the model to be applied during runtime. Outputting the generated model may also include storing the model in a data repository or other database. The model may then be retrieved by the computer system 150 and / or the user device during runtime.

[0074] As an illustrative example, the computer system 150 may receive image data of oranges under UV light. Using segmentation and analysis techniques, infection may be a feature identified by the computer system 150. Food items in the image data may be labeled as infected, and image data of oranges that do not show signs of infection may be labeled as good or healthy oranges. Using CNN, k-means clustering, or other machine learning techniques, an orange infection model may be trained to distinguish image data that show infection from image data that does not show infection. For example, the model may be trained to analyze each patch and / or pixel in the image data to determine whether an orange exists, and if an orange exists, determine whether the orange appears to show signs of infection based on the labeled image data. If the orange appears to show signs of infection, the model may be trained to mark or otherwise classify / label the orange in the image data as infected. The model may also be trained to label the orange with other descriptors for infection, including but not limited to numerical, Boolean, and / or string values ​​indicating the presence of infection in the orange.

[0075] Although the training in process 195 is described from the perspective of generating one model, process 195 can also be used to generate more than one model. As described herein, each model can be generated and trained to determine various other quality features in various other types of food items. In addition, although process 195 is described as being implemented by computer system 150, process 195 can also be implemented by one or more other computing systems, devices, device networks, and / or cloud-based systems.

[0076] Figure 1C is an illustration of an exemplary system 100 for evaluating the quality of one or more food items 102A-N based on image data. System 100 may include an image sensor 106, an extraction engine 110, a food item detection engine 120, a quality assessment engine 130, and a quality evaluation engine 140. For purposes of this disclosure, an "engine" may include one or more software modules, one or more hardware modules, or any combination thereof.

[0077] Image sensor 106 may be used to generate image data 108 representing attributes of food items 102A-N, where N is any positive integer greater than 0 and represents the number of food items 102A-N on conveyor belt 104. Image sensor 106 may be part of an imaging device, such as a Figure 1A The imaging device 160 is described. Figure 1C In the example of FIG. 1 , the image sensor 106 may be arranged such that the image sensor 106 is capable of capturing image data 108 representing one or more images of the food items 102A-N as the food items 102A-N advance along the conveyor belt 104, as described with reference to FIG. Figure 1A As described. In some implementations, the image sensor 106 may include one or more hyperspectral sensors configured to capture hyperspectral data representing the characteristics of the food items 102A-N. In such an implementation, each pixel of the hyperspectral image may correspond to a spectrum of infrared or ultraviolet light associated with the corresponding food item imaged by the camera, wherein the camera is equipped with one or more sensors operating within the corresponding spectrum range. The spectrum of visible light may also be used to reconstruct the RGB images of the food items 102A-N described throughout the present disclosure. In some implementations, the sensor 106 may be a low-resolution digital camera (e.g., 5M or lower), a high-resolution digital camera (e.g., 5MP or higher), or any other type of sensor that can capture image data 108. The sensor 106 may cover a spectral range of 300nm-1700nm. One or more other sensors (such as sensors in a hyperspectral camera) may cover wavelengths within the range of 300nm-2500nm. One or more other cameras may be used, including but not limited to the ultraviolet capture camera described herein.

[0078] In some implementations, the sensor 106 may include multiple sensors positioned at multiple angles relative to the food items 102A-N. For example, the sensor 106 may include a first camera and at least one additional second camera that capture image data 108 of the food items 102A-N from different viewing angles, respectively. In such a configuration, the one or more additional cameras may be used to generate the image data 108 based on wavelengths of light different from or additional to the wavelengths of light captured by the first camera. In general, any set of wavelengths of light may be obtained by the sensor 106.

[0079] Each particular camera among the one or more cameras can be configured to detect different or additional wavelengths of light in several different ways. For example, in some implementations, different sensors can be used in different cameras to detect different or additional wavelengths of light. Alternatively or additionally, each of the one or more cameras can be positioned at different heights, different angles, etc. relative to each other in an attempt to capture different wavelengths of light. In some implementations, one or more cameras can be at least partially positioned to capture portions of food items 102A-N that may be obscured from the perspective of the first camera.

[0080] In some implementations, one or more light sources 105 may be used to illuminate food items 102A-N so that image sensor 106 can clearly capture image data 108 of food items 102A-N, as described with reference to FIG. Figure 1AAs described. The light source 105 may include one or more light sources that produce the same or different electromagnetic radiation, respectively. In this example, the light source 105 is depicted as being fixed to the image sensor 106. In some implementations, the light source 105 may be positioned at one or more locations near the image sensor 106 so as to illuminate the food items 102A-N before and / or during the capture of the image data 108. In some implementations, the light source 105 may be selected based on the frequency of the electromagnetic radiation output. For example, in some implementations, the light source 105 may be a halogen light source. Alternatively or additionally, the one or more light sources 105 may be a diode or diode series of a broadband light emitting diode (LED), which may be used to provide light across a visible wavelength spectrum, a near infrared wavelength spectrum, an electromagnetic spectrum, or any other spectrum.

[0081] Light source 105 or a control unit of light source 105 may be communicatively connected to image sensor 106 or a control unit of image sensor 106. For example, image sensor 106 or a control unit of image sensor 106 may send a signal to light source 105 or a control unit of light source 105 to cause light source 105 to illuminate food items 102A-N at a specific power and / or with one or more specific wavelengths of light at a specific moment in time. In some implementations, the specific moment in time may be a predetermined amount of time before or during the capture of image data 108.

[0082] The image data 108 generated by the image sensor 106 can be provided as input to the extraction engine 110. The image data 108 can include one or more images of the food items 102A-N. Such images can also include one or more HSIs. In some implementations, the image sensor 106 can provide the image data 108 directly to the extraction engine 110. The extraction engine 110 can then process the image data 108. In some implementations, the image sensor 106 can store the image data 108 in a data repository and / or a memory device. Subsequently, the extraction engine 110 can access the data repository and / or the memory device to obtain and process the image data 108.

[0083] The extraction engine 110 can obtain / receive the image data 108. Figure 1CAs shown in, image data 108 can be an image of multiple food items 102A-N.Therefore, image data 108 can be made up of multiple images 108A-N, and each image 108A-N corresponds to food items 102A-N.Image data 108A-N can include the description of n food items 102A-N.Image data 108A-N can also include a part of the surrounding environment of food items 102A-N, such as a part of conveyor belt 104 or food processing facilities (for example, a tray, platform, container, box or other mechanisms for keeping food items 102A-N).As described herein, extraction engine 110 can process the image data 108 obtained, and extract a part 112 of image data 108.As shown in the figure, extracted part 112 can include food items 102A-N without surrounding environment.In some implementations, extracted part 112 can only include one of food items 102A-N.

[0084] The extracted image portion 112 of the image data 108 (referred to herein as the extracted image 112) can be provided as an input to the food item recognition engine 120. In some implementations, the extraction engine 110 can provide the extracted image 112 directly to the food item recognition engine 120. In some implementations, the extraction engine 110 can store the extracted image 112 in a memory device, which the food item recognition engine 120 can subsequently access.

[0085] The food item recognition engine 120 can use one or more object recognition algorithms / techniques to recognize the part corresponding to the food item 102A-N of the extracted image 112. As an illustrative example, the food item 102A-N can be avocado. The food item recognition engine 120 can be trained on the image of multiple avocados to determine whether one or more avocados are depicted from the extracted image 112, and which areas of the extracted image 112 include avocado. Therefore, the food item recognition engine 120 can be trained to determine the border around each food item 102A-N in the extracted image 112. The engine 120 can also be trained to produce output data in the form of the annotated image 122 of the food item 102A-N. In the process of producing the annotated image 122, the engine 120 can annotate or otherwise index each border 122A-N representing each food item 102A-N. As described throughout this disclosure, the machine learning trained model is then applied to each bounding box 122A-N to determine a quality metric for each food item 102A-N.

[0086] In some implementations, annotated image 122 may include a coordinate system to annotate or index the location of each food item 102A-N. Numerical values ​​(such as x and y values ​​in an x ​​and y coordinate system) may be used to represent the location of food items 102A-N in annotated image 122. Subsequent processing steps may use the numerical values ​​representing the location of food items 102A-N to determine the outer boundaries of each food item 108A-N.

[0087] The annotated image 122 generated by the food item recognition engine 120 can be provided as an input to the quality assessment system 130. In some implementations, the food item recognition engine 120 can provide the annotated image 122 directly to the quality assessment system 130. In other implementations, the food item recognition engine 120 can store the annotated image in a memory device, and the quality assessment engine 130 can subsequently access the memory device to obtain and process the annotated image 122.

[0088] The quality assessment engine 130 can be configured to determine the quality measure for each food item 108A-N depicted in the annotated image 122. As described throughout the present disclosure, the engine 130 can use one or more machine learning trained models to determine the quality measure. Each model can be trained to identify different features indicating different quality measures for food items 108A-N. For example, each model can be performed by different quality assessment engines 132A-N. Each quality assessment engine 132A-N can be configured to implement a specific quality assessment operation for the annotated image 122A-N of each food item 108A-N. For example, one of the quality assessment engines 132A-N can be configured to detect the presence of an infection in the annotated image 122A-N of the food item 108A-N. Another quality assessment engine 132A-N can be configured to detect a specific type of infection, such as a potential infection or a wound infection. Another quality assessment engine 132A-N can be configured to detect the hardness of the food item 108A-N in the annotated image 122A-N. Another quality assessment engine 132A-N can be configured to detect the maturity of the food items 108A-N in the annotated images 122A-N. In some implementations, one or more engines 132A-N can be executed in series. Sometimes, one or more engines 132A-N can be executed in parallel. Parallel execution can be beneficial to reduce the amount of time required for processing the annotated image 122 and implementing multiple quality assessments to the food items depicted therein. In some implementations, only some of these engines 132A-N can be selected for serial or parallel execution.

[0089] In some implementations, the set of quality assessment engines 132A-N can be configured based on the type of food items 108A-N being analyzed. The quality assessment engines 132A-N can also be configured based on the business practices of the business entity implementing the system 100 or the relevant stakeholders in the entire supply chain of the food items 108A-N.

[0090] Still refer to Figure 1C , the quality evaluation engine 140 can be configured to evaluate the overall quality of food items 108A-N based on the quality measure determined by engine 132A-N. For example, engine 140 can determine the aggregate quality of food items 108A-N in batches. Engine 140 can also determine the percentage of the surface area covered by the detected infection for each food item 108A-N. In some implementations, engine 140 can also determine the current edibility (e.g., consumption rate) of food items 108A-N and / or the edible time length of food items 108A-N (e.g., based on how many infections are detected in specific food items or food item batches, and / or based on the infection coverage percentage on specific food items or food item batches).

[0091] As described herein, the engine 140 can also determine one or more supply chain modifications based on the quality measure of the food items 108A-N. For example, if the engine 132A determines that the food items 108A-N are infected and the engine 140 determines that approximately 80-90% of the surface area of ​​each food item 108A-N (or most of the food items 108A-N) in the batch is covered with infection, the engine 140 can further trigger the update of the distribution plan for the food items 108A-N, so that the food items 108A-N are transported to the food processing plant, so that they will not be wasted. If the engine 140 determines that these food items 108A-N still have a certain remaining shelf life and are therefore consumable for a period of time, the engine 140 can generate instructions for transporting the food items 108A-N to a nearby retail environment, so that they can be purchased and consumed by end users before the shelf life / edibility expires. Many other exemplary operations may be determined and performed based on the quality metrics determined by engines 132A-N (eg, the engines may provide a quantitative means for distinguishing between different batches of food items based on aggregate quality scores).

[0092] As described throughout the disclosure, the quality metric generated by engine 132A-N can be a numerical value, a binary value and / or a Boolean value. The quality metric can be a numerical value on a predefined scale. The quality metric can also be values ​​such as "good", "very poor", "poor", "acceptable", "excellent", "moderate", "enough". As another example, the output data can be a vector of one of two different values ​​for each of the food items 108A-N depicted, which provides an indication (e.g., healthy or unhealthy / infected) that the quality is passed or the quality is not passed for each food item 108A-N. The overall quality metric can also be determined for each of the food items 108A-N depicted based on the vector of the quality metric output by engine 132A-N. One or more different values ​​can be defined based on the practice of the relevant stakeholders in the supply chain and / or the supply chain.

[0093] In some implementations, an overall quality score 134A-N may be generated for each food item 108A-N in the output image 134. The overall quality score 134A-N may be a numerical indication of how many infections there are in a particular food item 108A-N and / or how likely a particular food item 108A-N is to be infected. The higher the score on the numerical scale, the more likely an infection is to be present and / or the more infections are present in a particular food item. The lower the score on the numerical scale, the less likely an infection is to be present and / or the fewer infections are present in a particular food item. Figure 1C The numerical scale shown in may be on a scale of 0 to 1. One or more other numerical scales may be used including, but not limited to, 0 to 5, 0 to 3, 1 to 5, 1 to 3, 1 to 4, 0 to 100, etc.

[0094] The overall quality scores 134A-N can be viewed by relevant stakeholders in the supply chain. The stakeholders can view the scores 134A-N and determine one or more modifications to the supply chain. The quality metrics generated by the engines 132A-N can also be output in one or more other forms for review by relevant stakeholders.

[0095] Figure 2A -B is a flow chart of a process 200 for determining the quality of a food item based on image data. Process 200 may be implemented by computer system 150. Process 200 may also be implemented by one or more other computing systems, devices, computers, networks, cloud-based systems, and / or cloud-based services. For purposes of illustration, process 200 is described from the perspective of a computer system.

[0096] Reference Figure 2AIn process 200 of -B, the computer system may receive training image data in block 202. As described throughout, the image data may be captured under UV lighting conditions. Some of the image training data may be captured under other lighting conditions, such as under the visible spectrum and / or short-wave visible light. Capturing the image training data under other lighting conditions may require the use of at least one narrow bandpass filter between the food item being imaged and the imaging device so that the fluorescent signal from the food item is not washed out by the visible light from the light source. Figure 1A As described in , training image data can be received from an imaging device. Training image data can also be received from a database, data repository, or other repository storing image data of a specific food item. The image data can include an image of a specific food item, wherein its quality will be determined based on different features that are visible or invisible from the image data. Training image data for a food item can be captured at one or more different times throughout the supply chain of the food item. For example, training image data for a food item can be captured before it is transported from a farm to a storage facility, captured when it arrives at the storage facility, captured during transportation to a storage location in the storage facility, captured when it is transported from the storage facility to an end consumer retail environment, captured when it arrives at an end consumer retail environment, etc.

[0097] In box 204, the computer system can identify the food items in the training image data. The computer system can use object detection technology to generate a frame around each food item in the training image data. As described herein, a food item in the training image data can be used to train the model with high accuracy. A plurality of food items in the training image data can also be used to train, verify and / or test the model. The computer system can also generate a digital grid index for the frame, so that each grid index represents the different food items in the training image data. The grid index can be used to identify the food item in the data repository or other storage technologies.

[0098] Optionally, as part of identifying features, the computer system may apply ground truth labels to the image data. Features such as rot, mold, maturity, dehydration, bruises, etc. may be identified from the training image data for each food item in the image data and labeled / annotated accordingly. In some implementations, one or more features such as firmness and dry matter may not be visible from the training image data, and therefore may be labeled using other techniques, such as destructive techniques involving durometers and / or penetrometers that perforate or penetrate the skin, surface, cortex, or flesh of the food items in the training image data.

[0099] The computer system can also select food items in box 206. The computer system can use a random number generator to randomly select a frame from the training image data. The food items in the selected frame can be used for training purposes subsequently. In some implementations, the computer system can select a frame of food items including feature tags with a threshold value number. The computer system can also select a frame of food items that meet other threshold feature standards. The threshold feature standard can correspond to the feature of indicating an infection in a particular type of food item. For example, the computer system can select a frame of food items that include features to be modeled, including bruises, rot, mildew, dehydration, browning, other discoloration, etc., of the epidermis / surface / cortex of the food item.

[0100] The computer system can identify and cluster the food item features for the selected food item in box 208. As described herein, the computer system can analyze the pixel data in the frame of the selected food item, so that the pixel is distinguished and classified into each cluster. The first cluster can correspond to the pixel of the background in the frame. The second cluster can correspond to the pixel of the healthy part of the food item (such as the surface / epidermis / cortex of the food item does not include the part of the feature of being labeled as indicating infection). The third cluster can correspond to the pixel of the unhealthy part of the food item (such as the surface / epidermis / cortex of the food item includes the part of the feature of being infected such as bruises, mildew, rot, dehydration, discoloration, etc.) representing the food item. Additional or less clusters can also be generated in box 208. For example, the third cluster can be further decomposed into additional sub-clustering, to provide a more fine-grained analysis of the quality of the food item. As an illustrative example, the sub-clustering can correspond to the pixel of the center / origin of the mildew on the food item. Another sub-cluster may correspond to pixels representing the extent of the spread of mold on the surface of the food item emanating from a center / origin. As another example, the sub-clusters may correspond to different degrees of mold. One illustrative sub-cluster may be one that exhibits white mold, another may correspond to blue mold, another may correspond to potential infections that are not visible under visible light conditions but only present under UV light conditions, etc.

[0101] Optionally, in box 210, the computer system may map the clusters in a multi-color space. The multi-color space may be an RGB space. Mapping the clusters into the multi-color space may provide a clearer visualization of the different features represented in the training image data. Mapping the clusters into the multi-color space may also provide a clearer visualization of infections or other unhealthy features in food items that make the food items have lower quality at the current time and / or over a period of time. Such features may not be easily visible to the human eye in other cases or under the visible spectrum.

[0102] The computer system can train a model for detecting food item features in box 212. The computer system can use the clustering that is optionally mapped from box 210 to train the model. Model training can be implemented using k-means clustering technology. One or more other clustering techniques can also be used for model training. For example, Gaussian mixture model (GMM), K nearest neighbor (KNN), K median, hierarchical clustering and / or density-based spatial clustering (DBSCAN) technology with noise application can be used for model training. In some other implementations where pixels are not easily clustered into healthy / infected clusters, a supervised example segmentation model (e.g., based on CNN) can be trained. In this example, a training data set with labels can be generated, and different parts of the food item can be manually annotated. Subsequently, the model can be applied to each frame in the image data of the food item, and a segmentation map (e.g., a segmented image) can be predicted. Subsequently, the segmentation map can be used instead of (or in addition to) the clustering map. The segmentation map can be used in the same manner as clustering in the disclosed technology of processing 200.

[0103] The computer system may also train more than one model in block 212. The computer system may train a model for each food item type. The computer system may also train a model for each quality characteristic of a food item (e.g., maturity, firmness, infection, potential infection, wound infection, bruises, rot, mold, dehydration, sourness, flavor, etc.). The trained models may be stored in a data repository for retrieval during runtime use. The models may also be stored in local memory of the computer system for runtime use. For additional discussion of training models, see Figure 1B .

[0104] The model generated in box 212 can be trained to receive image data of food items (or batches of food items) as input. Using the model, the computer system can identify features indicating infection in the food items. The model can then return an indication of the presence of infection in the food items as output. The model output can be a numerical value, a Boolean value and / or a string value.

[0105] The numerical value may be, for example, a value on a predetermined scale. The predetermined scale may be, for example, 0 to 3.5. On this scale, a value closer to 0 may indicate that there is no or only a small amount of infection in the food item. A value closer to 3.5 may indicate that there is some amount of infection in the food item. As another example, a value closer to 0 may indicate that there is little or no possibility of infection, while a value closer to 3.5 may indicate that there is a higher possibility of infection in the food item.

[0106] Boolean values ​​can be binary, such as true / false, yes / no, 0 / 1, infected / not infected, etc. Boolean values ​​can indicate whether an infection is present in a food item. String values ​​can indicate whether an infection is present and / or the likelihood that an infection is present. For example, string values ​​output by the model can indicate "an infection is present in the food item," "an infection is not present in the food item," "an infection is likely to be present," and / or the extent to which an infection is present in the food item. The model can generate one or more other outputs.

[0107] During runtime, the computer system may receive image data of a food item (block 214). Figure 1A and 1C As described, image data can be captured under UV light conditions and received from an imaging device, such as imaging device 160 and / or image sensor 106. The food items in the image data can be of the same type as the food items in the training image data. In some implementations, the image data received in box 214 can be image data of only one food item. The image data can also include batches, platforms, trays, clusters or groups of food items of the same video type.

[0108] The computer system can apply a model to the image data in box 216 to determine that the infection in the food project exists. The computer system can determine that the infection in each food project in the image data exists. For example, the computer system can implement object detection technology to identify the border around each food project in the image data. The model can then receive each border as input to determine that the infection of the specific food project in the border exists. The computer system can also determine that the overall infection for the food project in the image data exists. For example, the model can receive the image data as input, and determine that the average infection for the food project exists. The model can also determine that the infection exists for each food project in the image data. Subsequently, the computer system can average, sum or otherwise aggregate for the infection of each food project to determine that the overall infection for the batches of food projects represented in the image data exists.

[0109] As described herein, a model can be used to analyze color features in image data. The colors in the image data can be clustered and mapped into a color space (e.g., RGB space, CIELAB space, or other (multiple) three-dimensional color spaces). The model can then analyze each cluster in the color space to determine whether an infection exists. For example, if a cluster associated with an infection has a pixel size and / or color value that exceeds a certain threshold infection standard, a computer system can determine that an infection exists. If a cluster associated with an infection is located in an area of ​​image data outside a boundary or demarcation of the image data (which can indicate the background of the image data), a computer system can also determine that an infection exists. The threshold infection standard can be based on historical data, such as customer preferences, the quality and / or level of infection of food items that consumers will still purchase and / or consume, and / or other factors. This process can be effective in accurately evaluating and scoring the infection features of food items, even if the features are not obvious under the visible spectrum. This process can also be easy and fast in the calculation method, thereby utilizing less time and computing resources to determine the quality of food items based on image data captured under UV light conditions.

[0110] In some implementations, the computer system can select which model or models will be applied to the image data based on the type of the imaged food item. For example, the computer system can use object recognition technology to identify the type of the food item in the image data. The computer system can also identify the type of food item by applying a recognition model to the image data. Machine learning techniques such as deep neural networks, CNNs can be used to train models to detect the food items in the image data and then identify the type of food items. The computer system can then retrieve one or more models associated with the type of the identified food items from the data repository or other databases storing the generated model. In some implementations, the computer system can use metadata as a part of the image data to determine which (which) models will be retrieved. The metadata, for example, can indicate the country of origin, geographic location of origin, type of food item, growing conditions, known / historical ripe conditions, etc. for food items.

[0111] In box 218, the computer system can determine whether an infection is detected in the food item based on the model output. As mentioned above, the model can be trained to generate an output indicating the presence of an infection in the food item. The computer system can return the output in box 218. The computer system can also normalize, process or otherwise convert the model output to an indication of the presence of an infection for a specific food item and / or batch of food items in the image data.

[0112] If no infection is detected, the computer system may return an indication of the healthy food item(s) (block 220). For example, if the model output indicates the presence of an infection below a predetermined threshold infection level, the computer system may determine that no infection is present in the food item. If the model output indicates the presence of no infection in the food item, the computer system may also determine that no infection is present in the food item. In some implementations, the computer system may determine that an infection is present if any number of infections are detected in the food item.

[0113] Returning the instructions regarding the healthy food item(s) may include storing the instructions in a data repository for later retrieval, use, and / or output. Returning the instructions may also include sending the instructions to a user device of a relevant stakeholder in the supply chain of the food item. The instructions may then be used by the computer system and / or the relevant stakeholder to determine, generate, and / or implement one or more supply chain modifications. For additional discussion of supply chain modifications, see Figure 3A -B.

[0114] If an infection is detected, the computer system may determine the percentage of infection coverage for the food item(s) in block 222. The computer system may make this determination for each food item that is determined to have an infection. To determine the percentage of coverage, the computer system may sum the number of pixels that indicate an infected portion of the food item and divide the sum by all pixels that indicate an infected portion and a healthy portion of the food item. The computer system may then multiply the result of the division by 100 to obtain a percentage value indicating how much of the total surface of the food item is infected.

[0115] In some implementations, a model may be trained in block 212 to determine the percentage of infection coverage in food items determined to be infected. Additionally, in some implementations, blocks 208 and 210 may not be implemented because the model may be trained to predict the percentage of infection coverage directly from the image data. Sometimes, the model may directly predict different segments in the image data, such as healthy, infected, and background segments, which are then used to calculate the percentage of infection coverage. In some implementations, the computer system may apply a model in block 222 that is generated and trained to determine the percentage of infection coverage for infected food items.

[0116] The computer system can return the indication of the percentage of infection existence and infection coverage in box 224. The computer system can return indication for each food item in the image data. The computer system can also return the aggregate value for batch food items (for example, by summing, averaging and / or aggregation, etc., for the total number of food items in the image data). The indication of the percentage of infection existence and infection coverage can be represented together with the image data. For example, the frame of the infected food item can be returned, wherein the food item is shown in a multi-color space (for example, RGB space). In the multi-color space, the infected part of the food item can be more clearly visualized from the healthy part of the food item and the background in the frame. The frame can also include the indication of the percentage of infection coverage for a specific food item.

[0117] As previously mentioned with respect to block 220, returning the indication in block 224 may include storing the indication in a data repository. Returning the indication in block 224 may also include sending the indication to a user device of an interested party for output in a graphical user interface (GUI) display. The indication may then be used by a computer system and / or an interested party to determine one or more supply chain modifications for a particular food item and / or batch of food items. For additional discussion of supply chain modifications, see Figure 3A -B.

[0118] Optionally, the computer system can determine and return the edible time length for food items in box 226. Given a sufficient number of time series data (e.g., 4 weeks) of infection evolution over time on food items, combined with metadata such as country of origin, harvest date, storage temperature, a model can be built to predict infection growth over time on food items, and thus estimate when infection will reach edible threshold. As an illustrative example, food items can be evaluated for fluorescence at time = 0. Fluorescence can be classified (e.g., none, low, medium, high) based on fluorescence signals. Food items can be cultivated under relevant conditions and track deterioration within a predetermined time amount, such as the following 4 weeks. The number of deterioration of each image within the predetermined time amount can be calculated subsequently. When new food items are analyzed, the fluorescence quantity detected for these new food items can be related / associated back to the model established from the raw data captured within the aforementioned 4 weeks. As a result, the computer system can estimate the deterioration risk in the new food items about future time points.

[0119] The determination in box 226 can be made for the following situations: for each food item in the image data, for each food item with infection in the image data, and / or as an average for the entire batch of food items in the image data. The computer system can apply an edible model to the image data of the food item identified as infected. The edible model can be trained to predict the edible length of time (e.g., shelf life, consumption, maturity, etc.) based on the presence of infection, the type of infection, the spread of infection, the percentage of infection coverage, and other information about a specific type of food item (e.g., origin, transportation conditions, historical edible conditions, historical maturity conditions, etc.). The edible length of time can indicate how long the food item remains available for consumption by the end consumer. Sometimes, a food item with an infection may still be edible / available for consumption for a period of time after the start time of the infection. The predicted edible length of time can be used by the computer system and / or relevant stakeholders to determine supply chain modifications to reduce potential waste of food items. For additional discussion on determining the edible length of time for food items, refer to Figure 4 .

[0120] Blocks 202-212 may be performed at a different time than blocks 214-226 (e.g., after a certain amount of time has passed between blocks 202-212 and blocks 214-226). For example, blocks 202-212 may be performed during a training phase. Blocks 214-226 may be performed at a later time as part of a run-time usage of the model(s) generated and trained during the training phase. In some implementations, blocks 214-226 may be performed immediately after blocks 202-212.

[0121] Figure 3A -B is a flow chart of a process 300 for determining the quality of a food item based on image data. Process 300 may be implemented by computer system 150. Process 300 may be implemented by one or more other computing systems, devices, computers, networks, cloud-based systems, and / or cloud-based services. For illustrative purposes, process 300 is described from the perspective of a computer system.

[0122] Reference Figure 3A In process 300 of -B, the computer system may receive image data of a food item in block 302. For further discussion, see Figure 1A , 1C and 2A-B.

[0123] In box 304, the computer system can implement object detection technology to identify the border around each food item in the image data. The computer system can utilize a food item analyzer and / or be trained to identify one or more machine learning models of food items that appear in the border. CNN and / or image classification models can be used to accurately identify the food items in the border. Each returned border (e.g., serial mode) can be processed in a separate job. In some implementations, the returned border can be processed in parallel. In some implementations, the computer system can identify the food items in advance by analyzing the metadata associated with the image data. Metadata can sometimes indicate, for example, what type of food items appear in the image data. The food item type can then be used by the computer system to retrieve the food item analyzer or other image classification models that are used to detect and identify each type of food item in the image data. In addition, if the food item does not appear in the image data, the border can not be returned for further processing and analysis.

[0124] The computer system can determine the grid structure based on the frame in box 306. The computer system can implement an index and assign an index to each frame that constitutes the grid structure of the image data. Each identified food item can receive one of the indexes, which can be used to identify the food item. In order to determine the grid structure, the computer system can (1) find out what looks like the most likely row of food items based on the Y height of the frame, and then (2) sort the entire data frame based on the X position. The grid can be determined, and an index value can be assigned to each frame in the grid structure, which can be used to identify the food items that appear in the frame. The model trained by one or more machine learning can also be used to determine the grid structure and assign an index.

[0125] Determining the grid structure and indexing the structure can help to associate the determined quality measure with the food items that appear in the image data. After all, each food item can have different quality measures (such as healthy, infected, mildewed, rotten, etc.), so each determined quality measure should be assigned to the index of the corresponding food item. In addition, for the food item assignment index value can help to be related to the additional measure and data determined by the computer system about the specific food item. As a result, the computer system can set up a more robust and accurate quality measure for the specific food item in the image data.

[0126] In some implementations, the grid structure can also be beneficial to facilitate searching all image data of food items to identify and / or output food items with specific features and / or quality metrics (e.g., output all food items with infection coverage percentages exceeding a certain threshold infection level). This can be beneficial to training the model described herein to more accurately identify and determine the quality characteristics of food items. This can also be beneficial to relevant stakeholders in the supply chain who are interested in monitoring food items, analyzing ROI, and / or making supply chain adjustments that help reduce or otherwise prevent waste based on food items.

[0127] The computer system may select a bounding box for the food item in the image data (block 308).The computer system may select one of the food items depicted in the image data to evaluate using the model(s) described herein.

[0128] The computer system can then apply clustering models such as k-means models to the selected frame in relaxation 310. The computer system can pass the k-means model of the quality of the food item based on pixel data (e.g., color) to the frame portion of the image data. Although the model is described as a k-means model, the model can also be any other model trained using clustering techniques as described herein. The model can return an indication about whether there is infection in the food item. The computer system can also select one or more additional / other models to be applied to the frame. Models can be selected based on the type of food item, user (e.g., stakeholder) preferences and / or any features (e.g., maturity, firmness, shelf life, edibility, edible time length, etc.) that may wish to evaluate the quality of the food item. Each model can be executed independently of each other. The model can be executed serially. In some implementations, the model can be executed in parallel.

[0129] Executing multiple models in box 310 can be beneficial to generate a robust and accurate quality assessment for food items. As an illustrative example, a first model can identify browning on the skin of a fruit. The first model can extract features in the image data that indicate browning. These extracted features can be provided as input to a second model, which can be trained to further identify the maturity stage of the fruit based on browning. The determination of the maturity stage of the fruit of the second model can be provided as input to a third model. The third model can be trained to determine an overall quality measurement score for the food item based on the maturity stage, wherein the maturity stage is further based on browning. Therefore, a more robust and non-invasive quality assessment can be implemented. It can be appreciated that any number of models can be used in any order to implement the quality assessment described herein.

[0130] The computer system can determine a quality measure of the food item based on the output from the clustering model (block 312). As described with reference to FIGS. 1-2 , the computer system can determine whether an infection exists in the food item. The computer system can determine a percentage of infection coverage for the food item. The computer system can also determine the current edibility of the food item and / or the length of time the food item can be eaten (e.g., remaining shelf life, remaining time available for consumption by an end user, etc.).

[0131] The computer system can also use the border grid index of food items to store the quality measure for food items in square frame 314. The computer system can associate the quality measure with the grid index assigned to the food items. The quality measure can be subsequently stored in a data repository or other types of databases using this association. As described herein, the quality measure can be retrieved by the computer system and presented at one or more user devices. The quality measure can be used in future analysis and monitoring for specific food items (or the batches of food items to which this specific food item belongs / is associated). In some implementations, the quality measure can also be used in future training data sets, to refine or improve the precision of the k-means model or other models described herein.

[0132] In block 316, the computer system may determine whether there are more food items in the image data. For example, the computer system may determine whether there are more index bounding boxes in the grid structure that have not yet been assigned a quality metric. In some implementations, the computer system may check a data repository to see which grid indexes have been assigned a quality metric to make the determination in block 316.

[0133] If there are more food items, the computer system can return to block 308 and repeat blocks 308-314 for each remaining food item in the image data. If there are no more food items in the image data, the computer system can continue to block 318.

[0134] In box 318, the computer system can output a quality measure for the food items in the image data. The quality measure can be presented in several ways. For example, the image or image data portion representing the food items can be used to describe the quality measure. The food items that are classified as infected can be represented by including a close-up image of a bruise or other feature on the food items that cause the poor quality of the food items and thereby the infected food items. As another example, a spectrogram can be used to describe the quality measure. As described herein, the quality measure can also be output as a numerical value, a Boolean value and / or a string.

[0135] In some implementations, the computer system can output one or more quality measures for one or more food items based on (multiple) user preferences. For example, a user at a user device can provide input to the user device, thereby requesting to view the quality measures for a subset of the food items depicted in the image data. The input can also request to view a subset of the quality measures for a specific food item represented in the image data. One or more other user inputs can be used to generate (multiple) customized / personalized quality measure outputs for display at a user device.

[0136] Optionally, the computer system may determine and return one or more supply chain modifications based on the quality measurement for the food item in the image data (box 320). The modification may include changing the location to which the food item is shipped, changing the amount of time the food item is stored, changing the storage conditions, applying a ripening agent or other treatment to the food item, discarding the food item, moving the food item for shipment to an end consumer, moving the food item for shipment to a food processing plant, etc. The supply chain modification may differ based on the quality measurement for the food item.

[0137] For example, if the food item is classified as being infected and / or the percentage of infection coverage exceeds the threshold infection level, then the computer system can determine that the food item should be shipped to the food processing plant and / or delivered to the grocery store that is geographically closest to the current position of the food item. Such supply chain modification can reduce the possibility that the food item is wasted. Similarly, if the edible time length predicted for the food item is less than the threshold time length, then the computer system can generate instructions to move the food item and / or the batched food items comprising the food item to the terminal consumer retail environment that is geographically closest to the current position of the food item, thereby the food item can be purchased and consumed by the terminal consumer before it is no longer edible / consumable.

[0138] As another example, if a food item is scored as healthy (e.g., good quality, does not include the presence of an infection, has an infection but the presence of the infection is less than a threshold infection level, etc.), the computer system can determine that the food item can be stored longer than other food items, and / or the food item can be shipped to a grocery store that is geographically further away from the current location of the food item. Based on what quality measure is determined for the food item, there may be one or more other modifications. Based on user-specified preferences associated with different quality measures evaluated by the (multiple) models described herein, there may also be one or more other modifications.

[0139] Figure 44 is a flow chart of a process 400 for determining the length of time a food item is edible based on its determined quality. A food item may still be edible when a small amount of infection is present (e.g., less than a threshold amount of yellow UV fluorescence is identified in the image data of the food item). Therefore, the process 400 can be used to predict how long a food item will be available for purchase and consumption since the time the infection was detected.

[0140] Process 400 may be performed by computer system 150. Process 400 may also be performed by one or more other computing systems, devices, computers, networks, cloud-based systems, and / or cloud-based services. For purposes of illustration, process 400 is described from the perspective of a computer system.

[0141] Reference Figure 4 In process 400 in block 402, a computer system may receive time series training image data of a quantity of food items identified as being infected. Image data may be captured at various points in time or predetermined time periods in the life cycle or supply chain of the quantity of food items. Image data may be captured daily (e.g., once a day) throughout the life cycle of the quantity of food items. Image data may also be captured at other predetermined time intervals during the life cycle of the quantity of food items. Image data may be captured under UV light settings. In some implementations, image data may also be captured under other light settings, such as under visible light conditions.

[0142] The computer system can train the edibility model to predict the edible time length of the food item based on the time series training image data (box 404). The time series training image data can show / track the development and / or growth of the infection in the food item over time. The time series image data can be labeled at different stages of infection development and / or growth. The different stages of infection development and / or growth can be related to other training data indicating the edibility (e.g., shelf life, freshness, maturity, consumption-ready state, etc.) of the food item. Other training data can be obtained using destructive techniques such as cutting, punching or pushing into the food item. Other training data can also be obtained using destructive techniques utilizing a penetrometer and / or a hardness meter. In addition, other training data indicating the edibility of the food item can include the taste, freshness, maturity, hardness, etc. of the food item during the life cycle of the food item.

[0143] The edibility model can then be trained to predict the edibility measure for food items using the labeled time-continuous training image data and other training data indicating edibility. The model can be trained using machine learning techniques such as CNN and / or deep neural networks. The edibility model can also be trained to identify the color of the infection indicating the food item in the image data, and then these colors are related to different edible levels and / or edible time lengths. For example, compared with image data with less than a threshold number of yellow UV fluorescence, image data with at least a threshold number of yellow UV fluorescence can represent a food item with a shorter remaining edible time amount. The yellow UV fluorescence can be related to the infection in the food item. The threshold number of the yellow UV fluorescence in the image data can also be different according to the type of the food item in the image data.

[0144] The edibility model can generate an output indicating the edible level of a food item. In some implementations, the model can also generate an output indicating the edible time length for a food item. The output can be a numerical value, a Boolean value, and / or a string value. For example, the model can generate a score indicating how the edible of a food item is. The score can be assigned a numerical value (e.g., a value on a scale of 1 to 100) on a predetermined scale. A value lower than a predetermined threshold value (e.g., a value between 1 to 50) can indicate that the edible of a food item is lower, or has a shorter remaining edible length. A value higher than a predetermined threshold value (e.g., a value between 51 to 100) can indicate that a food item is edible, or has a longer remaining edible time amount. The model can also generate an output indicating the predicted remaining edible time amount for a food item. For example, the output can include hours, days, weeks, etc., during which the predicted food item remains edible and available for consumption by end consumers.

[0145] Once the edibility model is generated and trained, the model can be stored in a data repository to be retrieved at a later time for use at runtime. The edibility model can also be stored in a local memory for use at runtime at the computer system.

[0146] During the run time, the computer system can receive the image data of the food item in square frame 406. The food item can belong to the same type as the food item in the time series training image data. In square frame 406, as described in Fig. 1-3, the computer system can only receive the image data for the food item that is identified as having infection. The image data can be the image data of a food item. The image data can also be the image data of batches of food items. In some implementations, the computer system can receive the image data for the food item, regardless of whether it is identified as having infection. Therefore, processing 400 can be implemented to determine the edible time length for healthy food items and infected food items or food items with other poor quality characteristics.

[0147] The computer system can apply an edible model to the image data in box 408 to determine the edible of the food item. The image data can be provided to the edible model as an input. The edible model can generate an output indicating whether the food item in the image data is edible and / or for the predicted edible time length of the food item. Therefore, as a part for determining the edible of the food item, the computer system can predict the edible time length (box 410) based on the edible of the food item. As described above, the edible model can be used to make the prediction. The edible time length can indicate the remaining time amount that the food item (or batches of food items) in the image data can consume for the end consumer.

[0148] The computer system can return the edible time length for the food item in box 412. The computer system can return the edible time length for the infected food items in the batch of food items (e.g., the food items with the highest degree of infection, or the food items with a quality measure that meets the threshold infection standard, such as the percentage of infection coverage). The computer system can also return the overall / average remaining edible time length for all food items in the batch. The computer system can also return the edible time length for each infected food item in the image data.

[0149] Optionally, the computer system may determine supply chain modifications based on the length of time that the food item or batch of food items can be consumed (block 414). For additional discussion of determining supply chain modifications, see Figure 3A - Process 300 in B.

[0150] Figure 5AAn exemplary infection detection analysis 500 using image data of a food item 501 under UV light is depicted. In this example, the food item 501 is a citrus fruit. As described throughout the present disclosure, one or more other food items may also be imaged, such as other citrus fruits. Using the disclosed techniques, UV light image data 502 of the food item 501 may be captured by an imaging device. The imaging device may capture images of a batch, tray, or platform of similar food items 501. The image data 502 may be a frame cropped from the image data of the batch, wherein the image data 502 depicts one food item 501 in the batch. As shown in the image data 502, the UV light conditions cause infected portions 503A and 503N of the food item 501 to fluoresce. The infected portion 503A fluoresces (e.g., yellow) in the image data 502, while the infected portion 503N indicates mold on the surface of the food item 501 (e.g., white). In the image data captured under conventional visible light settings, infected portions 503A and / or 503N may not be sufficiently visible or obvious. Conversely, infected portion 503A, for example, may be visible, but the extent of the spread of infected portion 503A on the surface of food item 501 may not be clearly visible.

[0151] Image data 502 may be processed by a computer system (e.g., computer system 150) by applying a model (e.g., a k-means clustering model, another model using a clustering algorithm, such as a Gaussian mixture model (GMM)) to identify clusters of pixels representing various features in image data 502. Model output may include processed image data 504, in which the features of image data 502 are clustered and mapped into RGB space. Each cluster in processed image data 504 may be shown with a color representing a different feature in image data 502.

[0152] For example, pixel cluster 508 may represent the background of image data 502. Another pixel cluster 510 may represent the healthy features of food item 501 from image data 502. Additional pixel clusters 512A and 512N may represent unhealthy or infected features of food item 501 from image data 502, such as infected portions 503A and 503N of food item 501, respectively. In this illustrative example, cluster 508 may be green, cluster 510 may be purple, cluster 512A may be yellow, and cluster 512N may be blue. In some implementations, one or more other different colors may be used to represent clusters 508, 510, 512A, and / or 512N.

[0153] like Figure 5AThe processed image data 504 shown in provides the clear visualization of the infection in the food item 501.Computer system as described herein can also generate infection coverage measurement 506.Infection coverage measurement 506 can show how many parts of visible food item 501 in image data 502 are covered by infection.Each side of food item 501 can be imaged as described herein to determine the overall infection coverage of the whole surface of food item 501.In some implementations, computer system described herein can also estimate / measure the infection of other sides of food item 501 that is not visually depicted in image data 502.For example, the computer system can estimate that the other side of food item 501 (being equivalent to the other half of the surface of the food item not shown in image data) is healthy, and thus calculates the infection coverage for the whole surface of food item 501.

[0154] Infection coverage metric 506 may be output or returned with processed image data 504 for presentation in a GUI display at a user device as described herein. Infection coverage metric 506 may be an average of pixels in infected clusters 512A-N shown in processed image data 504 relative to pixels in healthy clusters 510 and infected clusters 512A-N. Figure 5A In the example of , food item 501 has an infection coverage measure 506 of 0.182. This value, when multiplied by 100 to generate a percentage value, can correspond to an infection coverage of 18.2%. As described throughout the present disclosure, infection coverage measure 506 can also be presented as a percentage value.

[0155] Infection coverage measurement 506 shows how much surface of food item 501 is infected. Generally speaking, measurement 506 can be used to extrapolate that the inside of food item 501 may also be infected, and the inside of food item 501 may be more infected than the outer surface of food item 501. Some infections such as pathogens (e.g., penicillium) stay near the surface of food item 501, and therefore may not infect the inside of food item 501 too much. Therefore, the wound type signal in the image data usually shows surface damage for food item 501. On the other hand, potential infection usually moves through the inside of food item 501, and therefore spreads under the surface of food item 501. Therefore, the potential type signal in the image data usually shows that food item 501 can also include a certain level of internal damage.

[0156] Figure 5BAn exemplary infection detection analysis 550 using image data of a batch of food items under UV light is depicted. Here, image data 552 of a batch of oranges is captured under UV light conditions. As shown, various colors of fluorescence are emitted in each orange, indicating that a certain amount of infection may exist in each orange. The fluorescent color can indicate the center point of infection and the spread of infection on the surface of each orange. Processing the image data 552 using the techniques described herein can produce processed image data 554.

[0157] As described herein, image data 552 is processed and comprises another model of applying k-means clustering method or realizing clustering technology.Described model can generate outputs such as processed image data 554, wherein the clustering of feature (for example background, (multiple) healthy parts of food items, (multiple) infected parts of food items) is mapped in RGB space.Each distance in processed image data 554 can be presented with different colors.Computer system as described herein can determine how many food items in processed image data 554 have the color representing infection.Computer system can also determine the percentage of infection coverage for each food item in processed image data 554.In addition, computer system can determine or predict edible length for each food item in processed image data 554 and / or the whole batch of food items in processed image data 554.

[0158] Figure 6 is an exemplary system diagram of components used to determine food item quality based on the techniques described herein. Computer system 150 , food item quality data repository 620 , and model data repository 618 may communicate (eg, wired and / or wirelessly) via network(s) 180 .

[0159] Computer system 150 may include object detection engine 602 (e.g. Figure 1C Food item recognition engine 120, Figure 1C The extraction engine 110 and / or the combination of the food item identification engine 120 and the extraction engine 110 in the embodiment of the present invention), the indexing engine 604, the quality assessment engine 606 (e.g. Figure 1C 150, the quality assessment system 130, the quality evaluation engine 140, or a combination thereof), a supply chain modifier 607, a model training engine 608, and a communication interface 610. The communication interface 610 can provide communication between the components described herein. In some implementations, one or more of the components 602, 604, 606, 607, 608, and 610 can be separate from the computer system 150 and can be part of one or more other computer systems, computers, servers, devices, and / or networks.

[0160] Model training engine 608 can be configured to generate one or more models that can be used by quality assessment engine 606. Engine 608 can implement, for example, Figure 1B , Figure 2A - Blocks 202-212 of process 200 of B and Figure 4 The training described in blocks 402-404 in process 400 in the embodiment of the present invention. Engine 608 can generate a model for each quality metric / feature and / or for each food item type. Although the present disclosure describes a model for detecting the presence of infection and determining the edibility of a food item, one or more other models can also be generated using the same or similar techniques. The other models can include, but are not limited to, determining the browning, rotting, mildew, ripening, firmness, taste, acidity, Brix, size, shape, etc. of various types of food items, such as apples, avocados, berries, cucumbers, oranges, lemons, limes, other citrus fruits, etc.

[0161] The models generated and trained by the model training engine 608 can be stored as models 622A-N in the model data repository 618. The models 622A-N can be accessed and / or retrieved during runtime by one or more analyzers of the quality assessment engine 606. Based on the type of food items in the image data and / or user preferences for quality assessment of (multiple) specific food items in the image data, only some of the models 622A-N can be selected during runtime. The models 622A-N can also be updated or otherwise modified based on the runtime model application for the image data.

[0162] The object detection engine 602 can be configured to detect one or more food items in the image data as described herein, such as in Figure 3A -B described in the square frame 304 in the processing 300 of.Engine 602 can receive image data and implement object detection technology to process image data.Engine 602 can also apply the model that is trained to be trained to one or more machine learning trainings of any type of food project or specific type of food project in the recognition image data.Engine 602 can generate the frame around each food project in the image data.For each food project 624A-N, engine 602 can store the frame image of described food project in the food project quality data repository 620.

[0163] In some implementations, the object detection engine 602 can calibrate the colors in the border portion of the image data to maintain color consistency throughout the image data. Color calibration can also be performed by a separate engine, such as a color calibration engine (not shown). Color calibration can be applied to the entire image data as a pre-processing step prior to object detection and / or object extraction. Once the image data is calibrated based on color, RGB image analysis as described throughout the present disclosure can be performed.

[0164] Indexing engine 604 may be configured to apply a grid structure to the image data as described herein and index each bounding box in the structure, such as in Figure 3A -B processing 300 described in the square frame 306. Each frame can surround the food items represented in the image data.For each food item 624A-N, engine 604 can retrieve the frame image from the food item quality data repository 620, and the index value is assigned to the frame.Therefore, each food item can be identified by the grid index assigned, thereby can be of benefit to future information retrieval, analysis and processing operations.For each food item 624A-N, engine 604 can be stored in the grid index in the food item quality data repository 620.

[0165] The quality assessment engine 606 can be configured to identify one or more features that indicate the quality of the food item represented in the image data. As described herein, the engine 606 can determine the presence of infection in the image data. One or more other quality assessment engines can also be part of the computer system 150. Each quality assessment engine (e.g., Figure 1CThe quality assessment engines 132A-N in the food item can be configured to determine different quality characteristics of the food item. For example, the engine 606 can determine the presence of infection for any type of food item, another engine can determine the presence of infection for oranges, another engine can determine the ripeness of avocados, another engine can determine the firmness of avocados, another engine can be a universal color analyzer for any type of food item, another engine can be a size analyzer, another engine can be a color analyzer for apples, another engine can be a flavor and / or juiciness analyzer for oranges, etc. As another illustrative example, one or more quality assessment engines can include a color analyzer, an apple yellowing analyzer, a lime yellowing analyzer, an avocado interior analyzer, a lime rot and dehydration analyzer, a shelf life analyzer, a produce grade analyzer, a size analyzer, a shape analyzer, a ripening coating analyzer, a dry matter analyzer, a firmness analyzer, a Brix analyzer, a nutrient content analyzer, a sticker analyzer, an acidity analyzer, a universal internal defect analyzer, and / or a universal color analyzer. The one or more additional analyzers may include a wrinkle analyzer (e.g., for mangoes) and / or a pore oxidation (black spot) analyzer (e.g., for mangoes). The output from any one or more analyzers may be stored in the food item quality data repository 620 for each food item 650A-N as a quality metric (or quality score) for each food item 624A-N. In some implementations, the output from any one or more quality assessment engines may be received as input into one or more other quality assessment engines executed by the computer system 150 to determine the overall quality of a particular food item or a batch of food items containing the particular food item.

[0166] exist Figure 6 In some implementations, the quality assessment engine 606 can retrieve the bounding box image for each food item 624A-N from the food item quality data repository 620. In some implementations, the engine 606 can receive the bounding box image of the food item from the object detection engine 602. In some implementations, the engine 606 can directly retrieve the image data of the batch of food items from the imaging device or imaging sensor, such as Figure 1A The imaging device 160 and / or Figure 1C The imaging sensor 106 described in .

[0167] The bounding box of the food item can be processed by the infection analyzer 612. The infection analyzer 612 can be configured to determine whether there is an infection in the food item. The analyzer 612 can also determine whether there is an infection in the batch of food items based on analyzing the bounding box of each food item in the batch of food items. The analyzer 612 can retrieve one or more models 622A-N from the model data repository 618 to execute during runtime. For example, the analyzer 612 can retrieve the model for detecting the presence of infection described throughout the present disclosure. The analyzer 612 can also select a model from the model data repository 618 based on the food item type identification made by the object detection engine 602 and / or metadata associated with the bounding box of the retrieved food item (e.g., food item information for each food item 624A-N in the food item quality data repository 620, which may include origin, growing geographical conditions, transportation conditions, typical / historical maturity conditions, typical / historical growing conditions, food item type, etc.). The analyzer 612 can apply the model to determine whether there is an infection in the food item, such as in Figure 1A , Figure 2A -B and blocks 310-314 in process 300 of FIG. 3. The determination made by analyzer 612 may also be stored in food item quality data repository 620 for each food item 624A-N. As described herein, the determination may be a quality metric.

[0168] The quality assessment engine 606 can have an infection coverage determiner 614, which can be configured to determine how much of the food item in the bounding box is covered by the infection detected by the infection analyzer 612. The determiner 614 can generate a percentage value indicating the infection coverage for a particular food item, such as referring to Figure 2A -B's block 222 in process 200 and Figure 3A -B as described in block 312 of process 300. Determinator 614 may also generate one or more other values, such as a string value, a Boolean value, and / or a numeric value indicating how much of the food item is covered / includes an infection. The percentage of infection coverage or other values ​​of infection coverage may be stored in food item quality data repository 620 for each food item 624A-N.

[0169] The quality assessment engine 606 may also have an edible time length determiner 616, which may be configured to determine the current edibility (e.g., consumption rate) and / or the remaining time length that the food items in the predicted bounding box can be consumed / edible by the end consumer. In some implementations, the determiner 616 may also predict the shelf life and / or maturity of the food items. In addition, the determiner 616 may predict the overall edible time length for a batch of food items containing a particular food item in the bounding box. For additional discussion on determining the edible time length, see Figure 4 The edible time length may then be stored in the food item quality data repository 620 for each food item 624A-N.

[0170] Finally, the quality assessment engine 606 can generate an output regarding the quality of the food items analyzed by components 612, 614, and 616. The engine 606 can also generate an output regarding the overall quality of the batch of food items containing the analyzed food items. The output can be sent to Figure 1A (Multiple) user devices 170 depicted and described in .

[0171] Supply chain modifier 607 can be configured to determine one or more supply chain modifications for a particular food item or batch of food items based on the quality metrics determined by quality assessment engine 606. Modifier 607 can receive quality metrics (e.g., infection presence, infection coverage, and / or edible time length) from quality assessment engine 606 (or retrieved from food item data records 624A-N in food item quality data repository 620), and receive supply chain schedules for particular food items 624A-N from food item quality data repository 620. Modifier 607 can then determine one or more modifications to the supply chain schedule based on the quality metrics, wherein the modifications can ensure that waste of a particular food item is reduced or otherwise eliminated. For additional discussion of determining supply chain modifications, see Figure 1A , Figure 3A -B's block 320 in processor 300 and Figure 4 Block 414 of process 400 of . Any modifications or updates to the supply chain schedule may then be stored in the food item quality data repository 620 for each food item 624A-N.

[0172] Figure 7An example of a computing device 700 and an example of a mobile computing device that can be used to implement the techniques described herein are shown. Computing device 700 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Mobile computing devices are intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0173] The computing device 700 includes a processor 702, a memory 704, a storage device 706, a high-speed interface 708 connected to the memory 704 and a plurality of high-speed expansion ports 710, and a low-speed interface 712 connected to a low-speed expansion port 714 and the storage device 706. Each of the processor 702, the memory 704, the storage device 706, the high-speed interface 708, the high-speed expansion port 710, and the low-speed interface 712 is interconnected using various buses and can be mounted on a common motherboard or otherwise installed as appropriate. The processor 702 can process instructions for execution within the computing device 700, including instructions stored in the memory 704 or on the storage device 706, so as to display graphical information for a GUI on an external input / output device, such as a display 716 coupled to the high-speed interface 708. In other implementations, multiple processors and / or multiple buses can be used as appropriate along with multiple memories and multiple types of memories. Additionally, multiple computing devices may be connected, with each device providing some portion of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0174] The memory 704 stores information within the computing device 700. In some implementations, the memory 704 is one or more volatile memory units. In some implementations, the memory 704 is one or more non-volatile memory units. The memory 704 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0175] The storage device 706 can provide large-scale storage for the computing device 700. In some implementations, the storage device 706 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier. A computer program product can also include instructions that, when executed, implement one or more methods, such as those described above. A computer program product can also be tangibly embodied in a computer or machine-readable medium, such as a memory 704, a storage device 706, or a memory on a processor 702.

[0176] The high-speed interface 708 manages bandwidth-intensive operations for the computing device 700, and the low-speed interface 712 manages less bandwidth-intensive operations. Such functional allocation is merely exemplary. In some implementations, the high-speed interface 708 is coupled to the memory 704, the display 716 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 710 that can accept various expansion cards (not shown). In this implementation, the low-speed interface 712 is coupled to the storage device 706 and the low-speed expansion port 714. The low-speed expansion port 714 can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), which can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, for example, through a network adapter.

[0177] As shown in the figure, computing device 700 can be implemented in several different forms. For example, computing device 700 can be implemented as a standard server 720, or multiple times in a group of such servers. In addition, computing device 700 can be implemented in a personal computer, such as a laptop computer 722. Computing device 700 can also be implemented as part of a rack server system 724. Alternatively, components from computing device 700 can be combined with other components in a mobile device (not shown), such as mobile computing device 750. Each such device can contain one or more of computing device 700 and mobile computing device 750, and the entire system can be composed of multiple computing devices that communicate with each other.

[0178] The mobile computing device 750 includes, among other components, a processor 752, a memory 764, an input / output device such as a display 754, a communication interface 766, and a transceiver 768. The mobile computing device 750 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the processor 752, the memory 764, the display 754, the communication interface 766, and the transceiver 768 are interconnected using various buses, and several of the components may be mounted on a common motherboard or otherwise appropriately mounted.

[0179] The processor 752 may execute instructions within the mobile computing device 750, including instructions stored in the memory 764. The processor 752 may be implemented as a chipset including a plurality of separate analog and digital processor chips. The processor 752 may, for example, provide coordination of other components of the mobile computing device 750, such as control of a user interface, applications executed by the mobile computing device 750, and wireless communications of the mobile computing device 750.

[0180] The processor 752 can communicate with the user through the control interface 758 and the display interface 756 coupled to the display 754. The display 754 can be, for example, a TFT (thin film transistor liquid crystal display) display or an OLED (organic light emitting diode) display, or other suitable display technology. The display interface 756 may include suitable circuits for driving the display 754 to present graphics and other information to the user. The control interface 758 can receive commands from the user and convert them for submission to the processor 752. In addition, the external interface 762 can provide communication with the processor 752 to enable near-field communication of the mobile computing device 750 with other devices. The external interface 762 can provide, for example, wired communication in some implementations, or wireless communication in other implementations, and multiple interfaces can also be used.

[0181] The memory 764 stores information within the mobile computing device 750. The memory 764 may be implemented as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 774 may also be provided and connected to the mobile computing device 750 via an expansion interface 772, which may include, for example, a SIMM (Single In-line Memory Module) card interface. The expansion memory 774 may provide additional storage space for the mobile computing device 750, or may also store applications or other information for the mobile computing device 750. Specifically, the expansion memory 774 may include instructions for implementing or supplementing the processing described above, and may also include security information. Therefore, the expansion memory 774 may be provided, for example, as a security module for the mobile computing device 750, and may be programmed with instructions that allow secure use of the mobile computing device 750. In addition, a secure application may be provided through a SIMM card together with additional information, such as placing identification information on a SIMM card in an unbreakable manner.

[0182] The memory may include, for example, flash memory and / or NVRAM memory (non-volatile random access memory), as discussed below. In some implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, implement one or more methods, such as those described above. The computer program product may be a computer or machine readable medium, such as the memory 764, the expansion memory 774, or a memory on the processor 752. In some implementations, the computer program product may be received, for example, in a propagated signal via the transceiver 768 or the external interface 762.

[0183] The mobile computing device 750 can communicate wirelessly via a communication interface 766, which may include digital signal processing circuitry where necessary. The communication interface 766 may provide communications in various modes or protocols, such as GSM voice calls (Global System for Mobile Communications), SMS (Short Message Service), EMS (Enhanced Messaging Service) or MMS messaging (Multimedia Messaging Service), CDMA (Code Division Multiple Access), TDMA (Time Division Multiple Access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service) and others. Such communications may occur, for example, via a transceiver 768 using radio frequencies. In addition, short-range communications may occur, such as using Bluetooth, WiFi or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 770 may provide additional navigation and location-related wireless data to the mobile computing device 750, which may be used appropriately by applications running on the mobile computing device 750.

[0184] Mobile computing device 750 may also communicate audibly using audio codec 760, which may receive spoken information from a user and convert it into usable digital information. Audio codec 760 may likewise generate audible sounds for the user, such as through a speaker in an earpiece of mobile computing device 750. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on mobile computing device 750.

[0185] As shown in the figure, the mobile computing device 750 can be implemented in several different forms. For example, the mobile computing device 750 can be implemented as a cellular phone 780. The mobile computing device 750 can also be implemented as part of a smart phone 782, a personal digital assistant, or other similar mobile devices.

[0186] Various implementations of the systems and techniques described herein may be implemented in digital electronic circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from and send data and instructions to a storage system, at least one input device, and at least one output device.

[0187] These computer programs (also referred to as programs, software, software applications or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0188] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) that the user can use to provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and the input from the user can be received in any form, including acoustic, voice, or tactile input.

[0189] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser that a user can use to interact with implementations of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0190] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server occurs by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0191] Although this specification contains many specific implementation details, these details should not be interpreted as limiting the scope of the disclosed technology or the content that can be claimed, but as a description of the features of a specific embodiment that can be specific to the disclosed specific technology. Certain features described in the context of separate embodiments in this specification can also be implemented in a single embodiment in part or in combination as a whole. On the contrary, the various features described in the context of a single embodiment can also be implemented separately in multiple embodiments or in any appropriate sub-combination. In addition, although each feature may be described as working in a specific combination and / or initially claimed as such in this article, one or more features from the claimed combination can be deleted from the combination in some cases, and the claimed combination can be a variation for a sub-combination or a sub-combination. Similarly, although each operation may be described in a specific order, this should not be understood as requiring such operations to be implemented in the specific order or sequential order, or requiring all operations to be implemented to achieve the desired result. Specific embodiments of the subject matter of the present invention are described above. Other embodiments fall within the scope of the appended claims.

Claims

1. A method for determining contamination in a food item using image data, the method comprising: receiving, by a computing system, image data of a food item under ultraviolet (UV) light from an imaging device; performing object detection on the image data by a computing system to identify a bounding box surrounding each food item in the image data; determining, by a computing system, a grid structure for the image data based on a bounding box surrounding each food item in the image data, wherein each bounding box is assigned a grid index in the grid structure, the grid index being used to identify the food item in a data repository; determining, by the computing system, for each food item, a measure of the presence of infection for the food item by applying a model to a bounding box portion of the image data, wherein the model is trained using image training data of other food items under UV light, the image training data being annotated based on previously identifying a first portion of the other food items as having an infection characteristic and a second portion of the other food items as having a healthy quality characteristic, wherein the other food items are of the same food type as the food item; determining, by the computing system, an infection coverage measure for each food item based on determining that the infection presence measure for the food item indicates the presence of an infection; and The computing system returns, for each food item: (i) a bounding box portion of the image data containing the food item, (ii) a grid index, (iii) an infection presence measure for the food item, and (iv) an infection coverage measure for the food item.

2. The method according to claim 1, wherein: The infection coverage measure for a food item is the percentage of the surface of the food item that includes features indicative of infection.

3. The method according to any one of claims 1 to 2, wherein: The food item in the image data includes at least one of citron, mandarin orange, pomelo, orange, grapefruit, lemon, lime, and tangerine.

4. The method according to any one of claims 1 to 3, wherein: The infection presence measure is a string value indicating a healthy food item or an infected food item.

5. The method according to any one of claims 1 to 4, further comprising: identifying, by a computing system, a group of food items in the image data as having respective infection presence measures that satisfy a threshold infection level, the threshold infection level indicating that the food items are infected; determining, by a computing system, an edibility measure for the set of food items by applying an edibility model to the image data, wherein the edibility model is trained using training time series image data of infected food items annotated by prior identification of infected surface coverage associated with an edible length of time of the infected food items, wherein the infected food items are of the same food type as the food item; and An edibility measure for the group of food items is returned by the computing system.

6. The method according to any one of claims 1 to 5, wherein: The edibility measure indicates whether the group of food items is edible.

7. The method according to any one of claims 1 to 6, wherein: Determining, by the computing system, a edibility measure for the group of food items includes predicting an edible length of time for the group of food items.

8. The method according to any one of claims 1 to 7, wherein: The edibility measure indicates the length of time that the group of food items can be eaten.

9. The method according to any one of claims 1 to 8, wherein: The model is trained by a computing system using a process comprising the following steps: Receive image training data for other food items; Implement object detection on the image training data to identify training bounding boxes around food items; Identify healthy, infected, and background features of food items in the borders; Mapping the identified features into a multi-color space; as well as A model is trained using a clustering algorithm to determine an infection presence measure for a food item based on the identified features mapped into a multi-color space.

10. The method according to any one of claims 1 to 9, wherein: The clustering algorithm is k-means.

11. The method according to any one of claims 1 to 10, further comprising: The infection presence measure for each food item in the image data is sent by the computing system to the user computing device for display in a graphical user interface (GUI).

12. The method according to any one of claims 1 to 11, further comprising: retrieving, by the computing system, for each food item from a data repository, a measure of the presence of infection for the food item; identifying, by a computing system, supply chain information for the food item including an existing supply chain schedule and a destination for the food item; determining, by a computing system, whether to modify supply chain information for a food item based on the infection presence measure; In response to determining to modify the supply chain information, generating, by the computing system, modified supply chain information based on the infection presence measure, wherein the modified supply chain information includes one or more of a modified supply chain schedule and a modified destination for the food item; as well as The modified supply chain information is sent to one or more supply chain participants through a computing system to implement the modified supply chain information.

13. The method according to any one of claims 1 to 12, wherein: Determining, by the computing system, whether to modify supply chain information for the food item based on the infection presence measure includes determining that the infection presence measure satisfies a threshold infection level indicating that the food item is infected rather than healthy.

14. The method according to any one of claims 1 to 13, wherein: The modified supply chain information includes instructions that, when executed by one or more supply chain participants, result in the food item being moved for outbound shipment to an end consumer that is geographically proximate to the location of the food item.

15. The method according to any one of claims 1 to 14, wherein: The modified supply chain information includes instructions that, when executed by one or more supply chain participants, result in at least one of: (i) the food item being moved for outbound shipment to a food processing plant, or (ii) controlled atmospheric conditions surrounding the food item being altered.

16. The method according to any one of claims 1 to 15, wherein: The image data includes at least one of an RGB image, a hyperspectral image, a multispectral image, a nuclear magnetic resonance (NMR) image, and a magnetic resonance image (MRI).

17. The method according to any one of claims 1 to 16, wherein: Determining, by the computing system, for each food item, an infection presence measure for the food item includes determining that the food item is infected based on an infection cover of the food item exceeding a threshold infection cover.

18. The method according to any one of claims 1 to 17, further comprising: The food item is determined to be healthy based on the infection cover of the food item being less than a threshold infection cover.

19. The method according to any one of claims 1 to 18, wherein: Determining, by the computing system, an infection coverage measure for each food item based on determining that the infection presence measure for the food item indicates the presence of an infection includes: summing pixels in the image data representing healthy characteristics of the food item and unhealthy characteristics of the food item; Dividing the number of pixels in the image data that represent unhealthy characteristics of the food item by the summed pixels to generate an infection coverage value; and The infection coverage value is multiplied by a predetermined factor to generate a percentage value as a measure of infection coverage.

20. A system for determining the presence of an infection in a food item using image data, the system comprising: at least one light source configured to illuminate food items of the same food type, wherein the at least one light source emits ultraviolet (UV) light; one or more imaging devices configured to capture image data for the food item when the food item is illuminated by the at least one light source; as well as at least one computing system in communication with the one or more imaging devices, the at least one computing system being configured to: receiving image data of the food item under UV light from the one or more imaging devices; performing object detection on the image data to identify bounding boxes surrounding each food item in the image data; determining a grid structure for the image data based on a bounding box surrounding each food item in the image data, wherein each bounding box is assigned a grid index in the grid structure, the grid index being used to identify the food item in a data repository; determining, for each food item, a measure of the presence of infection for the food item by applying a model to a bounding box portion of the image data, wherein the model is trained using image training data of other food items under UV light, the image training data being annotated based on previously identifying a first portion of the other food items as having an infection characteristic and a second portion of the other food items as having a healthy quality characteristic, wherein the other food items are of the same food type as the food item; determining an infection coverage measure for each food item based on determining that the infection presence measure for the food item indicates the presence of an infection; and For each food item, return: (i) the bounding box portion of the image data containing the food item, (ii) the grid index, (iii) an infection presence measure for the food item, and (iv) an infection coverage measure for the food item.

21. The system of claim 20, wherein: The at least one light source is a UV LED lamp.

22. The system according to any one of claims 20-21, wherein: The at least one light source is a black light.

23. The system according to any one of claims 20 to 22, wherein: The at least one light source is a UV flood lamp.

24. The system according to any one of claims 20 to 23, wherein: The at least one light source emits UV light having a wavelength in the range of approximately 300 nm-400 nm.

25. The system according to any one of claims 20 to 24, wherein: The infection coverage measure for a food item is the percentage of the surface of the food item that includes features indicative of infection.

26. The system according to any one of claims 20-25, wherein: The food item in the image data includes at least one of citron, mandarin orange, pomelo, orange, grapefruit, lemon, lime, and tangerine.

27. The system according to any one of claims 20 to 26, wherein: The at least one computing system is further configured to: identifying a set of food items in the image data as having respective infection presence measures that satisfy a threshold infection level, the threshold infection level indicating that the food items are infected; determining an edibility measure for the set of food items by applying an edibility model to the image data, wherein the edibility model is trained using training time series image data of infected food items annotated by prior identification of infected surface coverage associated with an edible length of time of the infected food items, wherein the infected food items are of the same food type as the food item; and Returns an edibility measure for the set of food items.

28. The system according to any one of claims 20 to 27, wherein: The image data is time-series image data of the group of food items.

29. The system of any one of claims 20-28, wherein the at least one computing system is configured to determine the infection coverage measure for each food item based on determining that the infection presence measure for the food item indicates the presence of an infection based on the following steps: summing pixels in the image data representing healthy characteristics of the food item and unhealthy characteristics of the food item; Dividing the number of pixels in the image data that represent unhealthy characteristics of the food item by the summed pixels to generate an infection coverage value; and The infection coverage value is multiplied by a predetermined factor to generate a percentage value as a measure of infection coverage.

30. A non-transitory computer readable medium encoding instructions operable to cause a data processing apparatus to perform the operations of the method of any one of claims 1 to 29.

31. A method for determining the presence of an infection in a food item using image data, the method comprising: receiving, by a computing system, image data of a food item under UV light from an imaging device; determining, by a computing system, a measure of the presence of infection for the food item by applying a model to the image data, wherein the model is trained using image training data of other food items under UV light, the image training data being annotated based on previously identifying a first portion of the other food items as having an infection characteristic and a second portion of the other food items as having a healthy quality characteristic, wherein the other food items are of the same food type as the food item; and An infection presence measure is returned by the computing system for the food item in the image data.

32. The method of claim 31 further comprising: determining an infection coverage measure for the food item based on the presence of an infection indicated by the infection presence measure by a computing system; as well as An infection coverage measure is returned by the computing system for the food item in the image data.

33. The method according to any one of claims 31-32, wherein: Returning, by the computing system, the infection presence measure for the food item in the image data includes sending the infection presence measure and the infection coverage measure to the user device for presentation in a GUI display at the user device.

34. A non-transitory computer readable medium encoding instructions operable to cause a data processing apparatus to perform the operations of the method of any one of claims 31 to 33.

Images