Ultraviolet Light and Machine Learning-Based Evaluation of Food Quality

Abstract

The disclosed technology provides for determining an infection within a food by using image data of the food under ultraviolet (UV) light. The method includes performing object detection on the image data to identify a surrounding bounding box for each food within the image data; applying a model to the bounding box portion of the image data to determine an infection presence metric for each food, the model being trained by using image training data of other foods under UV light, the image training data being annotated based on previous identifiers of a first portion of other foods having infection characteristics and a second portion of other foods having healthy quality characteristics; and determining a food infection coverage metric based on a determination that the infection presence metric for each food indicates the presence of an infection.

Description

Technical Field

[0001] Cross - Reference to Related Applications

[0001] This application claims the benefit under 35 U.S.C. § 119(e) of U.S. Patent Application No. 63 / 328,052, filed Apr. 6, 2022, entitled “ULTRAVIOLET LIGHT AND MACHINE LEARNING - BASED ASSESSMENT OF FOOD ITEM QUALITY,” which is hereby incorporated by reference in its entirety for all purposes.

[0002]

[0002] This specification describes devices, systems, and methods related to determining the quality of food, for example, based on image data of the food.

Background Art

[0003]

[0003] Foods such as agricultural products, fruits, and meats can have various quality metrics that can affect their suitability for consumption and value within the supply chain. Multiple different stakeholders across the supply chain are interested in evaluating such food quality metrics. As an example, food color can be an indicator of quality that can be used to sort and grade food. Various quality metrics can also influence consumer purchasing decisions.

[0004]

[0004] Foods with quality metrics such as good color characteristics can be more valuable than foods with bad color or other poor quality metrics. Color and other quality metrics can be used to indicate food ripeness, hardness, infection, spoilage, dryness, flavor, sweetness, and sourness characteristics. Any of these characteristics can be valuable across the supply chain and in consumer consumption decisions. For example, browning of food can indicate spoilage or infection. Early detection of browning color can enable supply chain corrections to be made to avoid wasting the food.

[0005]

[0005] It can be difficult to objectively and quantitatively define high-quality food based on image data of food in the visible light spectrum. Stakeholders in the supply chain can observe and compare colors or features that are visible within or on the food. However, the human eye can distinguish color differences or other feature differences only in extreme cases or when the color differences or other feature differences are significantly obvious. For example, the human eye can distinguish when a lime is yellow, green, or brown, or when a lime is more yellow than other limes, but the human eye may not be able to distinguish more subtle changes in color. As another example, it is difficult for the human eye to universally grade and compare all possible colors that a particular food can acquire over its lifespan. Subtle changes in color and other features over the lifespan of the food can represent changes in the quality of the food.

Summary of the Invention

[0006]

[0006] This specification generally describes systems, methods, and techniques for non-invasively evaluating the quality of food (e.g., vegetables, fruits, meat), for example, based on image data. Specifically, the disclosed technology provides for evaluating the quality of food from image data captured under or by ultraviolet (UV) light conditions. Sometimes, features of food indicating quality (e.g., subtle changes in the color of areas surrounding damage, spoilage, or other infections) may not be readily distinguishable in the visible light spectrum. UV light can cause fluorescence in potential infections such as damage, spoilage, drying, and mold. Since potential infections appear more visibly in UV light, image data can be captured from food under UV light conditions. The image data can include not only images (e.g., RGB, hyperspectral, multispectral, etc.) but also additional metadata. As a specific example, the disclosed technology can be used to evaluate food color to determine food quality such as the presence of infection, ripeness, rapidity for consumption, mold, spoilage, drying, etc. The disclosed technology can distinguish color differences (whether readily obvious or subtle) to assist in determining one or more different quality metrics for the food.

[0007]

[0007] For example, the disclosed technology can identify the presence of infection in citrus fruits (such as oranges, limes, lemons, etc.), what surface area of the citrus fruit is covered by the infection, the current level of edibility of the citrus fruit based on the presence of infection, and the predicted length of time the citrus fruit remains edible. The disclosed technology can make such determinations regarding a single food item and / or a batch of foods of the same food type. At least one model can be trained by machine learning techniques such as k-means clustering to determine the above-described quality metrics regarding foods. At least one model can also be trained to identify various other quality metrics (such as shelf life, ripeness, spoilage, mold, drying, damage, bruising, etc.) regarding various other types of foods (such as avocados, apples, berries, mangoes, cucumbers, chili peppers, etc.). As a result, the quality of foods (such as damage, infection, age, ripeness, taste, etc. or other non-visible characteristics of food quality) can be evaluated, especially when the human eye may not be able to provide objective and universal quantification or may not be able to distinguish subtle changes in the food. Further, different types of infections (such as latent infection, wound infection) have different fluorescence signatures. Each of these unique signatures can be identified by using a model trained by machine learning techniques (such as k-means clustering) to identify such types of infections with high precision and high throughput. Based on the detected infection, the disclosed technology can also determine and predict the shelf life of the food and / or the length of time the food remains edible so that appropriate supply chain modifications can be made to eliminate or reduce food waste.

[0008]

[0008] The disclosed technology can provide for generating various metrics that can be used to identify various characteristics of food indicative of such quality. Quality metrics can be defined on a per-food basis. Quality metrics can further be defined on a per-food-type basis. For example, apples may have different color metrics and corresponding machine learning trained models to identify infections or other quality metrics / characteristics compared to limes, avocados, oranges, lemons, limes, and other fruits and crops.

[0009]

[0009] The disclosed technology can be used to determine the presence of infection within food. Based on the determined presence of infection, corrections can be made early to the supply chain. For example, the quality assessment described throughout this disclosure can be made when the food enters a storage facility. The quality of the food, which can be an important indicator of how long the food can remain edible or consumable, can be evaluated at this point (e.g., even if the food has some presence of infection). If the quality of the food is identified as poor (e.g., if the food has an amount of infection exceeding a certain threshold health level), the supply chain can be corrected to immediately ship the food to consumers in the nearest geographical location (e.g., if the disclosed technology determines that the food can still be consumed for a certain amount of time) (either to discard the food or to ship the food to a food processing factory). If the quality of the food is identified as good (e.g., if the food has no infection or has an amount of infection lower than the threshold health level), the supply chain can be corrected to store the food for a period of time or to transport the food to consumers far from the geographical location. One or more other supply chain corrections can be determined based on the preferences of stakeholders throughout the supply chain lifecycle and the determined quality metrics of the agricultural product.

[0010]

[0010] One or more embodiments described herein may include a method for determining an infection in food by using image data. The method includes receiving, by a computing system, image data of food under ultraviolet (UV) light from an imaging device; performing object detection on the image data by the computing system to identify a surrounding bounding box for each food in the image data; determining, by the computing system, a grid structure of the image data based on the surrounding bounding box for each food in the image data, wherein each bounding box is assigned a grid index within the grid structure, and the grid index is used to identify the food in a data store; determining, by the computing system, an infection presence metric for each food by applying a model to the bounding box portion of the image data, wherein the model is trained by using image training data of other foods under UV light, and the image training data is annotated based on previous identifiers of a first portion of other foods having infection characteristics and a second portion of other foods having healthy quality characteristics, and the other foods are of the same food type as the food; determining, by the computing system, an infection coverage metric for the food based on a determination that the infection presence metric for each food indicates the presence of an infection; and returning, by the computing system, for each food, (i) the bounding box portion of the image data including the food, (ii) the grid index, (iii) the infection presence metric for the food, and (iv) the infection coverage metric for the food.

[0011]

[0011] In some implementations, the embodiments described herein may optionally include one or more of the following features. For example, the infection coverage metric for the food may be a percentage of the surface of the food that includes features indicating an infection. The food in the image data may include at least one of a citron, a mandarin, a pomelo, an orange, a grapefruit, a lemon, a lime, and a tangerine. The infection presence metric may be a string value indicating a healthy food or an infected food.

[0012]

[0012] The method also includes identifying, by a computing system, a group of foods in the image data, each having an infection presence metric that meets a threshold infection level, where the threshold infection level indicates that the food is infected; determining, by the computing system, an edibility metric for the group of foods by applying an edibility model to the image data, where the edibility model is trained by using training time series image data of infected foods, the training time series image data being annotated with previous identifiers of infection area coverage related to the length of time of edibility of the infected foods, the infected foods being of the same food type as the food; and returning, by the computing system, the edibility metric for the group of foods. The edibility metric may indicate whether the group of foods is edible. Determining, by the computing system, the edibility metric for the group of foods may include predicting the length of time of edibility of the group of foods. The edibility metric may indicate the length of time of edibility of the group of foods.

[0013]

[0013] As another example, the model is trained by a computing system by using a process that includes: receiving image training data of other foods, performing object detection on the image training data to identify training bounding boxes around the foods, identifying health features, infection features, and background features of the foods within the bounding boxes, mapping the identified features into a multi-color space, and training the model by using a clustering algorithm to determine an infection presence metric of the foods based on the identified features mapped into the multi-color space. The clustering algorithm can be the k-means method. In some implementations, the method can include transmitting, by the computing system to a user computing device, an infection presence metric for each food in the image data for display within a graphical user interface (GUI).

[0014]

[0014] This method may also include: retrieving from a data store, for each food, a food's infection presence metric by a computing system; identifying, by the computing system, food supply chain information including the food's existing supply chain schedule and destination; determining, by the computing system, whether to modify the food's supply chain information based on the infection presence metric in response to a determination to modify the supply chain information; generating, by the computing system based on the infection presence metric, modified supply chain information, wherein the modified supply chain information includes one or more of a modified supply chain schedule for the food and a modified destination; and transmitting, by the computing system, the modified supply chain information to one or more supply chain stakeholders to implement the modified supply chain information. Determining by the computing system whether to modify the food's supply chain information based on the infection presence metric may include determining that the infection presence metric meets a threshold infection level, wherein the threshold infection level indicates that the food is not healthy and is infected. The modified supply chain information, when executed by one or more supply chain stakeholders, may include an instruction to move the food to the end consumer geographically closest to the location of the food for shipment. The modified supply chain information, when executed by one or more supply chain stakeholders, may include an instruction to cause at least one of (i) moving the food to a food processing facility for shipment, or (ii) causing a change in the controlled atmospheric conditions surrounding the food.

[0015]

[0015] As another example, the image data may include at least one of an RGB image, a hyperspectral image, a multispectral image, a nuclear magnetic resonance (NMR) image, and a magnetic resonance image (MRI). Further, determining, by a computing system, a food infection presence metric for each food includes determining that the food is infected based on the infection coverage of the food exceeding a threshold infection coverage. The method may also include determining that the food is healthy based on the infection coverage of the food being less than the threshold infection coverage.

[0016]

[0016] In some implementations, determining, by a computing system, a food infection coverage metric for each food based on a determination that the infection presence metric for the food indicates the presence of an infection may include: summing the pixels in the image data representing the healthy characteristics and the unhealthy characteristics of the food; dividing the amount of pixels in the image data representing the unhealthy characteristics of the food by the summed pixels to generate an infection coverage value; and multiplying the infection coverage value by a predetermined coefficient to generate a percentage value as the infection coverage metric.

[0017]

[0017] One or more embodiments described herein may include a system for determining the presence of infection in food by using image data, the system comprising: at least one light source capable of illuminating food of the same food type, the at least one light source emitting ultraviolet (UV) light; one or more imaging devices capable of capturing image data of the food when the food is illuminated by the at least one light source; and at least one computing system in communication with the one or more imaging devices, the at least one computing system being configured to: receive image data of the food under UV light from the one or more imaging devices; perform object detection on the image data to identify the surrounding bounding boxes for each food in the image data; determine a grid structure of the image data based on the surrounding bounding boxes for each food in the image data, wherein each bounding box is assigned a grid index within the grid structure, and the grid index is used to identify the food in a data store; determine an infection presence metric for each food by applying a model to the bounding box portion of the image data, wherein the model is trained by using image training data of other foods under UV light, and the image training data is annotated based on previous identifiers of a first portion of other foods having infection characteristics and a second portion of other foods having healthy quality characteristics, and the other foods are of the same food type as the food; determine an infection coverage metric for each food based on the determination that the infection presence metric for each food indicates the presence of infection; and for each food, return (i) the bounding box portion of the image data containing the food, (ii) the grid index, (iii) the infection presence metric of the food, and (iv) the infection coverage metric of the food.

[0018]

[0018] This system may optionally include one or more of the following features. For example, at least one light source may be a UV LED light. At least one light source may be a black light. At least one light source may be a UV floodlight. At least one light source may emit UV light having a wavelength in the range of about 300 nm to 400 nm. The food infection coverage metric may be the percentage of the surface of the food that includes features indicating infection. The food within the image data may include at least one of lemon, mandarin, pomelo, orange, grapefruit, lime, and tangelo.

[0019]

[0019] In some implementations, at least one computing system may also: identify a group threshold infection level of the food within the image data, assuming each infection presence metric that meets a threshold infection level, where the threshold infection level indicates that the food is infected; determine a group edibility metric of the food by applying an edibility model to the image data, where the edibility model is trained by using training time series image data of infected food, the training time series image data is annotated with previous identifiers of infection surface coverage related to the length of time of edibility of the infected food, and the infected food is of the same food type as the food; and return the group edibility metric of the food. The image data may be time series image data of a group of food.

[0020]

[0020] As another example, at least one computing system can determine, based on a determination that an infection presence metric for each food item indicates the presence of an infection, and a food infection coverage metric is based on: summing pixels in the image data representing the healthy characteristics and the unhealthy characteristics of the food item, dividing the amount of pixels in the image data representing the unhealthy characteristics of the food item by the summed pixels to generate an infection coverage value, and multiplying the infection coverage value by a predetermined coefficient to generate a percentage value as the infection coverage metric.

[0021]

[0021] One or more embodiments described herein may include a method for determining the presence of an infection in a food item by using image data, the method comprising receiving, by a computing system and from an imaging device, image data of the food item under UV light, determining, by the computing system, an infection presence metric for the food item by applying a model to the image data, the model being trained by using image training data of other food items under UV light, the image training data being annotated based on previous identifiers of a first portion of other food items having infection characteristics and a second portion of other food items having healthy quality characteristics, the other food items being of the same food type as the food item, determining, and returning, by the computing system, the infection presence metric for the food item in the image data.

[0022]

[0022] This method may optionally include one or more of the following features. For example, this method may include determining, by a computing system, an infection coverage metric for food based on an infection presence metric indicating the presence of an infection, and returning, by the computing system, an infection coverage metric for food in the image data. In some implementations, returning, by the computing system, an infection presence metric for food in the image data may include sending the infection presence metric and the infection coverage metric to a user device for presentation within a GUI display on the user device.

[0023]

[0023] The devices, systems, and techniques described herein may provide one or more of the following advantages. For example, the disclosed techniques may provide for accurately assessing food quality characteristics that may not be easily detectable within the visible light spectrum, either via image data or by the human eye. Accordingly, image data of food may be captured under several UV light conditions. UV light may cause fluorescence in potential infections such as bruises, spoilage, drying, mold, etc. Since potential infections appear more visibly in UV light, image data captured under such conditions may be used to more accurately assess food quality than under other lighting conditions.

[0024]

[0024] Further, by using a machine learning trained model, the food quality can be more accurately determined from the subtle differences that appear in the image data captured under UV light conditions. The human eye can be prone to errors when trying to observe subtle changes in the appearance of food, and thus may not be able to detect non-visible features of food such as potential infection or wound infection. For example, the human eye may not be able to notice a slight discoloration on the orange-colored portion surrounding a mold spot (which may indicate an early sign of spoilage or other poor quality characteristics such as shortening the length of time for edibility / consumption). Further, human operators would need to be trained to visually measure the color and quality of food in a quantitative manner. This can be a time-consuming way and can thus be influenced by human prejudice. The disclosed technology provides for automatically and accurately detecting various quality features within food from a high-quality image-labeled image dataset and / or an unlabeled image dataset. The disclosed technology provides for analyzing the quality of food beyond just the visible spectrum and color, which may all be difficult or impossible for the human eye to observe and analyze. Thus, the disclosed technology can provide a deeper analysis by increasing efficiency and reducing human errors that may occur from observing the visual features of food.

[0025]

[0025] As another example, the disclosed technology can be used to make appropriate supply chain modifications early enough within the supply chain life cycle to reduce food-based waste. Food quality can be evaluated at any point throughout the supply chain. For example, quality can be evaluated before the food is shipped from the farm to the storage facility. Quality can also be evaluated when the food arrives at the storage facility. In some implementations, quality is even evaluated when the food is being marketed at the grocery store and becomes available to consumers. When food quality is evaluated early within the supply chain life cycle, the food can be sorted more appropriately based on the identification and / or predicted quality of such food. For example, food identified as being of good quality (e.g., without any potential infection or wound infection) when entering a storage facility can be stored within the facility for a longer period of time than food identified as being of poor quality (e.g., having a certain amount of potential infection or wound infection). Improved decisions can also be made regarding how and when to handle the food. For example, if one or more foods are determined not to meet a quality threshold (e.g., having a threshold amount of infection present), the disclosed technology can determine that an appropriate supply chain modification would be to initiate the application of an antibacterial treatment to the food. The consumption and edibility time frames, which can affect when the food is delivered to the grocery store (e.g., how long the food stays within the storage facility, whether the food should be delivered to a food processing plant instead of the grocery store), can also be determined by using the disclosed technology.

[0026]

[0026] As consistently described, the disclosed technology can generate a robust quality assessment of food. Various models can be generated and trained by using machine learning techniques having a high-quality image-labeled training data set to identify and score various quality characteristics associated with various foods. Such a robust quality assessment can be advantageous for more precisely monitoring food quality and for modifying the supply chain accordingly to reduce or otherwise eliminate food-based waste.

[0027]

[0027] As yet another example, the disclosed technology provides for evaluating the quality of food in a non-destructive manner. Since the model is trained to analyze the quality characteristics of food from image data under UV light conditions, it may not be necessary for a human to perform destructive techniques such as piercing or pressing on the skin, surface, or flesh of the food to determine those qualities. As a result, food delivered to the end consumer can be accurately tested and evaluated for quality without actually degrading the quality of such food. Thus, the model is trained to extract quality features from the image data of the food instead of requiring a human to destroy or otherwise alter the food before it is delivered to the consumer. Higher quality food can be delivered to the consumer, and the food may not be wasted when the disclosed technology is used to evaluate food quality.

[0028]

[0028] Details of one or more implementations are set forth in the accompanying drawings and the description below. Other features and advantages will be apparent from the description and drawings, and from the claims.

Brief Description of the Drawings

[0029]

Figure 1A

[0029] It is a conceptual diagram for determining food quality based on image data in ultraviolet (UV) illumination.

Figure 1B

[0030] It is a conceptual diagram for generating a model to determine the quality of food.

Figure 1C

[0031] It is a diagram of an exemplary system for evaluating the quality of one or more foods based on image data.

Figure 2A

[0032] It is a flowchart of a process for determining food quality based on image data.

Figure 2B

[0032] It is a flowchart of a process for determining food quality based on image data.

Figure 3A

[0033] It is a flowchart of a process for determining the quality of food based on image data.

Figure 3B

[0033] A flowchart of a process for determining the quality of food based on image data.

Figure 4

[0034] A flowchart of a process for determining the length of time of food edibility based on their determined quality.

Figure 5A

[0035] Describes an exemplary infection detection analysis using image data of food in UV illumination.

Figure 5B

[0036] Describes an exemplary infection detection analysis using image data of a batch of food in UV illumination.

Figure 6

[0037] An exemplary block diagram of components used to determine food quality based on the techniques described herein.

Figure 7

[0038] A schematic diagram showing an example of a computing device and a mobile computing device.

DETAILED DESCRIPTION OF THE INVENTION

[0030]

[0039] Like reference numerals in the various figures indicate like elements.

[0031]

[0040] This specification generally relates to techniques for evaluating the quality of one or more foods from image data captured under UV light conditions. The UV light conditions can reveal damage, infection, or other poor / unsound quality characteristics within the food that may not be visible or apparent under visible light conditions. UV-A induced fluorescence (such as light having wavelengths in the range of 300 nm to 400 nm) can cause unhealthy quality characteristics within the food to fluoresce and thus be made more visible for accurate quality evaluation of such foods. The disclosed techniques can provide for acquiring image data of one or more foods under UV light conditions and determining a quality metric for each food represented within the image data based on an evaluation of the image data. The model can be trained by using machine learning techniques to process the image data and determine the quality metric. The quality metric can include the presence of an infection within the food. Based on the determined presence of an infection, the disclosed techniques can also provide for determining the shelf life of the food, the length of time the food is edible (even if the food has the presence of an infection), and other quality metrics. One or more additional or other quality metrics can also be modeled and determined based on the food and / or food type and by using image data of the food under UV light conditions. Further, the disclosed techniques can provide for modifying one or more supply chain operations based on an evaluation of the quality of the food in an effort to mitigate losses that may result from foods having a quality level that does not meet certain thresholds.

[0032]

[0041] Referring to the accompanying drawings, FIG. 1A is a conceptual diagram for determining food quality based on image data in ultraviolet (UV) illumination. The computer system 150, the imaging device 160, and the user device 170 may be in a communication (e.g., wired and / or wireless) state via the network 180. The computer system 150 may be configured to evaluate the quality of the imaged food (such as agricultural products) as described throughout the present disclosure (e.g., refer to FIGS. 1C and 6). The imaging device 160 may include an image sensor 106. The imaging device 160 may be a camera that captures still images, time-series video data, and / or video data. The imaging device 160 may be an RGB camera that mainly captures light within the visible range. Any other camera designed to capture visible light (including, but not limited to, cameras of mobile devices (such as smartphone cameras, tablet cameras, laptop cameras, computer cameras, etc.) and professional or non-professional cameras (such as DSLR cameras)) may be used. UV fluorescence imaging with emitted light having a peak centered at a wavelength of about 520 nm may be used by the disclosed technology. The imaging device 160 may be used in automatic mode to capture images of the foods 102A - N under UV light conditions. The imaging device 160 may also be manually set for low-light settings. For example, the shutter speed of the imaging device 160 may be 1 / 8, its aperture may be set to F / 3.5, and its ISO may be set to 6400. One or more other camera settings may be used to capture images of the foods 102A - N under the UV light conditions described throughout the present disclosure.

[0033]

[0042] At least one light source 105 may also be configured to be in communication with the imaging device 160 (e.g., refer to FIG. 1C). In some implementations, the light source 105 may be separate from the imaging device 160. The light source 105 may be positioned near the imaging device 160 to provide uniform illumination over the area where the imaging device 160 captures images of the foods 102A - N.

[0034]

[0043] The light source 105 can be a UV emission light source. For example, the light source 105 can emit light having a wavelength within the range of 300 nm to 400 nm. As another non-limiting example, the light source 105 can be a UV-B light source that emits light having a wavelength within the range of about 200 nm to 350 nm (e.g., 280 nm to 315 nm). As another example, the light source 105 can be a UV-A light source that emits light having a wavelength within the range of about 300 nm to 420 nm (e.g., 315 nm to 400 nm). The light source 105 can also emit light having a wavelength within one or more other non-limiting exemplary ranges (including, but not limited to, 500 nm to 555 nm, 520 nm to 570 nm, and 420 nm to 620 nm). In some implementations, for example, the light source 105 can emit light having a wavelength of 365 nm. In some implementations, the light source 105 can be a UV LED light, a UV LED black light, a high-performance LED bulb, and / or a UV floodlight. For example, the light source 105 can be an LED UV light within 50 W. The light source 105 can also be a low LED UV light of about 4 W that is used in combination with an imaging device set to a manual or other specific camera setting for picking up fluorescence signals from the foods 102A to N. A plurality of light sources 105 can also be positioned around the imaging device 160 to provide uniform illumination conditions. As described herein, the UV light interacts with the surfaces of the foods 102A to N and induces fluorescence (within the visible light range). The output of the imaging device 160 can be a normal RGB image regarding how the foods 102A to N look under illumination from the UV light (highlighting some defects such as infections).

[0035]

[0044] In some implementations, one or more narrow-bandpass filters can be used between the UV light source 105 and the foods 102A - N being illuminated. Such filters can help reduce visible light emission and target specific excitation wavelengths. These filters can target not only, for example, 285 nm, 330 nm, 365 nm, 395 nm, or 400 nm, but also any other wavelength of interest. The narrow-bandpass filter can also be used between the illuminated foods 102A - N and the imaging device 160 such that only the light emitted by the phosphors of the foods 102A - N (e.g., fluorescent compounds emitted by the foods 102A - N under UV light conditions) is captured. These filters can also target wavelengths between 400 nm and 780 nm and can specifically target 520 nm to assist in accurately capturing an image of the foods 102A - N since infections within the foods 102A - N fluoresce under UV light conditions. The imaging device 160 and the light source 105 can be positioned at various locations along the supply chain of the foods 102A - N. For example, the imaging device 160 and the light source 105 can be placed within a warehouse or other storage facility along a conveyor belt 104 system that transports the foods 102A - N to various locations within the warehouse. The imaging device 160 and the light source 105 can also be positioned within a closed environment (e.g., a pallet of batches 102A - N of foods, or a flat) that houses the foods 102A - N, such as a photo box, so that uniform lighting and environmental conditions, as described in U.S. Patent Application No. 63 / 295,172, titled "MACHINE LEARNING-BASED ASSESSMENT OF FOOD ITEM QUALITY," which is hereby incorporated by reference in its entirety, enable "accurate and uniform images to be captured." The photo box can be a closed unit that eliminates illumination from sources other than the light source 105 (such as ambient light). Imaging performed by the photo box can be advantageous for determining the risk of damage over a given time frame for a batch of foods 102A - N placed therein. Further, two or more images can be captured from the foods 102A - N inside the photo box to capture the entire surface of the foods 102A - N.As a result, the disclosed technology can be applied to multiple images of foods 102A - N to determine the overall presence of infection across the entire surface of each of the foods 102A - N, not just one side of the foods 102A - N.

[0036]

[0045] In some implementations, a section of the conveyor belt 104 having the imaging device 160 and the light source 105 can be covered within some materials or within some types of enclosures such as a photobox to avoid the intrusion of visible light. For example, two sets of rubber - style flaps or other materials can be positioned at the entry and exit points of the enclosure around the said section of the conveyor belt 104. The rollers of the conveyor belt 104 can be activated while the foods 102A - N are inside the enclosure so that they complete one full rotation by the time the foods 102A - N are delivered by the moving conveyor belt 104 from the enclosure. By rotating the foods 102A - N while they are inside the enclosure, the entire surface of each of the foods 102A - N can be imaged. The disclosed technology can be used with such images to determine the presence of infection across the entire surface of each of the foods 102A - N.

[0037]

[0046] By using the enclosure, images of the foods 102A - N can be captured within the covered portion of the conveyor belt 104 without the intrusion of visible light. Wherever the image is captured by the imaging device 160, the inner surface of the above - mentioned enclosure or photobox can also interfere with the reflection of the fluorescence signal captured in the image by the imaging device 160, so it can be coated with a certain material to limit the reflection. The materials used in making / constructing the enclosure or photobox can be selected so as not to fluoresce at the excitation wavelength of the light emitted by the light source 105.

[0038]

[0047] In some implementations, the imaging device 160 can be a portable device such as a mobile phone or a tablet that can be used by a human user to capture images of the foods 102A - N. The light source 105 can be attached to the imaging device 160 such that the light source 105 can illuminate the foods 102A - N as the portable device is moved across the foods 102A - N. As depicted in FIG. 1A, the imaging device 160 and the light source 105 can be positioned on the conveyor belt 104 within the storage facility.

[0039]

[0048] The imaging device 160 can be configured to continuously capture (block A) image data of the foods 102A - N as it is moved along the conveyor belt 104 to one or more storage locations or other destinations within the storage facility. The images can be captured under UV light conditions provided by the light source 105.

[0040]

[0049] The foods 102A - N can include, but are not limited to, fruits, citrus fruits, vegetables, and meats. For example, the foods 102A - N can include fruits of citrus plants (including, but not limited to, Citrus medica, Citrus reticulata, Citrus maxima, and hybrids thereof (such as oranges, grapefruits, lemons, limes, and tangerines)). As another example, the foods 102A - N can include other types of fruits and agricultural products (including, but not limited to, apples, mangoes, avocados, cucumbers, etc.).

[0041]

[0050] Here, the food items 102A - N are received from a shipper at a storage facility and can be placed on the conveyor belt 104. The food items 102A - N can be in cases, containers, on pallets, and on flats, and / or can be placed directly on the conveyor belt 104. In some implementations, the food items 102A - N can be stationary on the conveyor belt 104 within the storage facility or, alternatively, can be in motion when an image is captured by the imaging device 160. For example, some of the food items 102A - N can be randomly sampled as described in U.S. Patent Application No. 63 / 295,172: titled “MACHINE LEARNING - BASED ASSESSMENT OF FOOD ITEM QUALITY” and placed inside a photo box. A camera located inside the photo box can capture image data of the food items 102A - N under uniform UV illumination conditions, and the image data can then be analyzed by the computer system 150 to obtain an estimate of the quality distribution of the batch of food items 102A - N.

[0042]

[0051] The imaging device 160 can transmit the image data to the computer system 150 (block B). Next, the computer system 150 can apply one or more models to the image data to identify features of the food items 102A - N (block C). The features can indicate the quality of the food items 102A - N (such as whether the food items 102A - N are healthy or infected). The features identified by using the models can include various colors or textures on the surface, peel, or rind of the food items 102A - N that correlate with the healthy quality metric and the infected quality metric of that particular type of food. The models can be trained to identify and categorize the fluorescence color features of the food items 102A - N within the image data that have various quality features indicating the health of the food items 102A - N (such as browning, spoilage, damage, drying, mold, bruising, potential infection, wound infection, etc.).

[0043]

[0052] Accordingly, in block C, computer system 150 may apply a model to the image data to identify various color features of foods 102A - N indicative of the health of the foods 102A - N. The model may be trained by using machine learning techniques to identify features of various foods. Computer system 150 may select one or more models to apply in block C based on the types of foods 102A - N identified within the image data. Further, as described throughout the present disclosure, computer system 150 may apply the model to each of the foods 102A - N identified within the image data to determine a per - food quality metric (e.g., health, infection, spoilage, mold, etc.). Computer system 150 may also apply the model to a batch or flat of foods 102A - N captured within the image data or flat to determine the overall quality metric of the batch or flat of foods 102A - N.

[0044]

[0053] In block D, computer system 150 may determine the quality metric of foods 102A - N based on the identified features. The quality metric may be the presence of an infection. Accordingly, in block C, computer system 150 may determine, based on the output from the model, whether an infection is present within a single food of foods 102A - N and / or within a batch of foods 102A - N within the image data. For example, if computer system 150 determines that a food 102A - N is infected, computer system 150 may also, in block C, use the image data and / or the output from the model to determine the percentage of the surface of each food covered by the infection. Computer system 150 may also determine the current edibility (e.g., shelf life, consumption rating, freshness, ripeness, etc.) of foods 102A - N based on whether an infection is present. Further, computer system 150 may determine / predict the length of time for which foods 102A - N are still edible or, if not, still likely to be good for consumption. Such determination / prediction may be useful for determining one or more supply chain modifications of foods 102A - N as described herein.

[0045]

[0054] The output from the model in block C can be a boolean value (e.g., Yes / No, true / false, 0 / 1, etc.) indicating whether an infection is present. The output can also include a numerical value on a predetermined scale, where the numerical value indicates the likelihood that an infection is present within foods 102A - N (e.g., 0 on a scale of 0 - 100 can indicate that foods 102A - N are healthy and no infection is present, and 100 can indicate that an infection is present in large quantities on foods 102A - N / there is a large amount of infection in foods 102A - N). In some implementations, the output of the model can include several indicators of whether an infection is present within each of the foods 102A - N. In block D, next, computer system 150 can aggregate the indicators to determine whether an infection is present with respect to the entire batch of foods 102A - N.

[0046]

[0055] In block D, computer system 150 may normalize the output from the model to determine the quality metrics of foods 102A - N. For example, computer system 150 may adjust the output to an infection categorization scale indicating damage or infection of foods 102A - N. The scale may be a numerical scale including, but not limited to, 0 - 3.5, 0 - 4, 1 - 3, 1 - 5, 1 - 4, etc. A lower number on the scale indicates that foods 102A - N are healthy and there is little to no presence of infection (e.g., the presence of infection is below a threshold infection level that may be specific to the type of food or other characteristics related to the food and / or the growing / origin conditions of the food) or no presence of infection. A higher number on the scale indicates that foods 102A - N are unhealthy / infected (e.g., a high amount of damage within a particular food, batch 102A - N of foods, etc.) and there is a presence of infection (e.g., the presence of infection exceeds the threshold infection level). By using the disclosed technology, computer system 150 may identify various types of infections (such as blue mold, green mold caused by Penicillium sp.) that are often indicated as distinct ring-like symptoms emanating from the location of the wound in the food. The disclosed technology may also be used to detect potential types of infections such as Diplodia sp., Colletotrichum sp., and Alternaria sp. Each of these infections may have various types of symptoms (e.g., droplets emanating from the stem end or stem button of the food) that the model may be trained to detect, identify, and / or quantify to determine the quality metrics of foods 102A - N. Further, as described herein, the model may be trained to detect these various types of infections in various types of foods (including, but not limited to, citrus fruits, agricultural crops, vegetables such as cucumbers and peppers, avocados, apples, berries) and other types of foods that may exhibit such pathogens and infections.

[0047]

[0056] Computer system 150 may transmit quality metrics of foods 102A - N to user device 170 (block E). For example, computer system 150 may transmit each quality metric for each of foods 102A - N within the image data. Computer system 150 may transmit only some quality metrics of some of foods 102A - N. Computer system 150 may also, in block E, transmit all of the quality metrics of the batch or flat of foods 102A - N.

[0048]

[0057] In some implementations, computer system 150 may also transmit quality metrics to a database for storage. The quality metrics may be stored along with other historical measurements and additional metadata associated with the batch of foods 102A - N and / or the individual foods within the batch. This stored information may be used within a feedback loop for the continuous improvement and training of machine - learning trained models used to perform the techniques described herein. For example, by using previously determined quality metrics, one or more high - level models may be improved and / or trained to identify quality metrics that depend on seasonality, diversity, size, country of origin, and other factors.

[0049]

[0058] User device 170 may output quality metrics in block F. User device 170 may be a mobile device, smartphone, tablet, laptop, or other computer that can be used by stakeholders within the supply chain. In some implementations, the stakeholders may be supply chain parties. Stakeholders can examine the quality metrics for each of foods 102A - N (and / or their batches) to understand or analyze the return on investment (ROI) of foods 102A - N. The output metrics can also be used by stakeholders to monitor the quality of foods 102A - N over time and optionally make one or more supply chain modifications based on the current and / or predicted quality of foods 102A - N (e.g., edibility, consumer grading, ripeness, hardness, infection, etc.).

[0050]

[0059] Optionally, user device 170 may determine one or more supply chain modifications based on the output quality metrics of foods 102A - N (block G). User device 170 may automatically determine or otherwise recommend one or more supply chain modifications for foods 102A - N based on their corresponding quality scores. In some implementations, computer system 150 may determine or otherwise recommend supply chain modifications and send those recommendations to user device 170. Stakeholders at user device 170 may optionally execute, modify, or reject any of the recommended supply chain modifications. In some implementations, stakeholders may review the output quality metrics and determine the supply chain modifications to be implemented.

[0051]

[0060] Blocks A - G were described in FIG. 1A in connection with determining quality metrics for the presence of infection within food, but blocks A - G may also be performed to determine one or more other quality metrics of one or more other types of food. The one or more other quality metrics may include, but are not limited to, the quality of the skin of the food and / or whether the skin is intact (which may be indicative of resistance to infection and / or mold). The determination regarding the skin may further be used to determine whether the food is damaged and / or whether the wound is infected. The disclosed technology may also be used to determine whether the food is infected by a potential infectious pathogen. Further, although blocks A - G were described in connection with determining the presence of infection within a batch of food, blocks A - G may also be performed to determine the presence of infection within an individual food.

[0052]

[0061] FIG. 1B is a conceptual diagram of process 195 for generating a model for determining the quality of food. As described throughout the present disclosure, models of various foods, various types of foods, and various characteristics associated with a particular food may be generated.

[0053]

[0062] Computer system 150 may receive food image data 190 at block A. The image data 190 may include digital RGB images, hyperspectral images, and / or multispectral images depicting a particular food, food type, various foods, various food types, a single food, and / or a batch of food. The food image data 190 may include images of food captured under UV light conditions as described throughout the present disclosure.

[0054]

[0063] The image data 190 may also include images, tables, and / or other data of specific foods having some specific features to be modeled (such as spoilage and drying), and images of the same type of foods without the specific features to be modeled. For example, the food image data 190 may include a table stored in a data store that is extracted from an image of a food and labeled and / or annotated (e.g., automatically by a computer system (such as the computer system 150) and / or manually by a stakeholder or other user) with features (such as spoilage, drying, probability of determining shelf life, etc.). In some implementations, the food image data 190 may include images of the exterior and / or interior of the food. In some implementations, the food image data 190 may include images of a specific food at various stages of ripeness and between some stages of ripeness. The image data 190 may also include images of a specific food at various stages of infection progression (such as browning, spoilage, mold growth, drying, damage, etc.) over a certain period of time.

[0055]

[0064] The food image data 190 can be a robust set of training data that shows a plurality of different features that can exist and / or develop for a specific food over the entire life cycle of the food. The image data 190 can also be a set of images of the same food from various angles so that the entire food can be sufficiently analyzed by using the techniques described herein. Further, in some implementations, the image data 190 can include labels for the features, states, and / or qualities of the food. In some further implementations, such features, states, and / or qualities of the food can be learned by using the image data 190 without labels.

[0056]

[0065] As described above, the computer system 150 receives the image data 190 (block A). The image data 190 can be received from one or more imaging devices such as the imaging device 160 described herein. The image data 190 can also be retrieved from a data store.

[0057]

[0066] In block B, computer system 150 can identify and cluster the characteristics of food from the image data. In some implementations, computer system 150 can perform object detection techniques to identify each food item within image data 190. Next, computer system 150 can randomly select one of the identified food items for training purposes. Accordingly, computer system 150 can generate and train the model described herein by using one of the food items within image data 190. In some implementations, computer system 150 can generate and train the model described herein by using two or more of the food items within image data 190.

[0058]

[0067] In block B, computer system 150 can identify features indicative of the quality of the food. For example, computer system 150 can identify the color of the food correlated with / corresponding to spoilage, mold, various types of texture, bruising, or other types of infection from RGB image data. As further described throughout this disclosure, the identified features can be labeled. Object detection techniques can be performed on the image data to detect the food item, identify the food item, and then perform additional extraction steps to pick out one or more specific features of the food item.

[0059]

[0068] Computer system 150 can also cluster food characteristics based on their color data in block B. Computer system 150 can use the k-means clustering technique to cluster characteristics based on their colors. Computer system 150 can also use other types of clustering techniques. Computer system 150 can generate three clusters. One cluster can represent the background in the image data 190 (e.g., the flat, pallet, container, conveyor belt or other surface on which the food is placed). Another cluster can represent the healthy characteristics of the surface on the food in the image data 190 (e.g., the skin or peel of the food that does not include colors indicating infection, spoilage, mold, drying, browning, damage, etc.). Another cluster can represent the unhealthy or infection characteristics on the surface of the food in the image data 190 (e.g., the skin or peel of the food that includes colors indicating infection, spoilage, mold, drying, browning, damage, etc.). In some implementations, computer system 150 can generate fewer or more clusters. For example, computer system 150 can generate one or more additional clusters to represent one or more other characteristics of the food for determining food quality.

[0060]

[0069] In block C, computer system 150 can map the clusters of food characteristics into a multi-color space. The multi-color space can be the RGB space. One or more other color spaces can be used for mapping purposes. In block C, computer system 150 can map the cluster of infection characteristics, the cluster of healthy characteristics, and the cluster of background characteristics for various colors into the multi-color space.

[0061]

[0070] Next, computer system 150 may generate a machine learning model by using the mapped clusters (block D). In some implementations, the model may be generated and / or trained by one or more other computing systems, computers, a network of devices, and / or cloud-based services. The model may be trained, for example, by a remote computer system, stored in a data store, and accessible and executable by computer system 150. The model may be generated by using machine learning techniques including, but not limited to, k-means clustering or other clustering techniques. Convolution neural networks (CNNs) and other machine learning techniques may also be used for training purposes. Computer system 150 may generate a model for each feature of the identified and labeled features. Each model may also be trained to score the quality of the food based on the identified features. As an example, the model may be trained to determine the presence of infection within an orange and score the amount of the presence of infection within the orange.

[0062]

[0071] For example, computer system 150 can search for a flat orange image data. Computer system 150 can process the image data within the RGB space. By using object detection technology, computer system 150 can identify each orange within the image data and generate a bounding box around each orange. Computer system 150 can select one of the bounding boxes (e.g., randomly) to analyze the orange within the RGB space. Computer system 150 can train a model by using the k-means clustering technique to identify and cluster color features indicating the background, healthy part, and unhealthy or infected part of the orange. The clusters can be visualized within the RGB space to accurately identify the center of infection, the spread of infection on the surface of the orange, the healthy part (non-infected part) of the surface of the orange, and the background surrounding the orange. The model can also be trained to output an indicator of whether an infection is present within the orange. The indicator can be a boolean value (e.g., true / false, Yes / No, unhealthy / healthy, infected / uninfected, etc.), a string value (e.g., "infected food"), and / or a numerical value (e.g., on a scale of 0 - 5, where 0 indicates healthy / uninfected and 5 indicates most unhealthy / most infected).

[0063]

[0072] Next, the generated model can be output by computer system 150 (block E). During runtime, the generated model can be applied to the image data to identify features indicating the quality within the imaged food. Outputting the generated model can include presenting the model to a user on a user device. Next, the user can select the model to be applied during runtime. Outputting the generated model can also include storing the model within a data store or other database. Next, the model can be retrieved during runtime by computer system 150 and / or the user device.

[0064]

[0073] As a specific example, computer system 150 may receive image data of oranges within UV illumination. By using segmentation and analysis techniques, infection may be a characteristic identified by computer system 150. While food within the image data may be labeled as infected, image data of oranges showing no signs of infection may be labeled as good oranges or healthy oranges. By using CNN, k-means clustering, or other machine learning techniques, an orange infection model may be trained to differentiate image data showing infection from image data not showing infection. For example, the model may be trained to analyze each patch and / or pixel within the image data to see if an orange is present, and, if an orange is present, to analyze based on labeled image data to see if the orange appears to show signs of infection. If the orange appears to show signs of infection, the model may be trained to tag the orange within the image data as infected or otherwise classify / label it. The model may also be trained to tag the orange by other descriptors of infection (including but not limited to numerical, boolean, and / or string values indicating the presence of infection within the orange).

[0065]

[0074] Training in process 195 is described from the perspective of generating one model, but process 195 may also be used to generate two or more models. Each model may be generated and trained to determine various other quality characteristics within various other types of food as described herein. Further, while process 195 is described as being performed by computer system 150, process 195 may also be performed by one or more other computing systems, devices, networks of devices, and / or cloud-based systems.

[0066]

[0075] FIG. 1C is a diagram of an exemplary system 100 for evaluating the quality of one or more food products 102A-N based on image data. The system 100 may include an image sensor 106, an extraction engine 110, a food detection engine 120, a quality assessment engine 130, and a quality assessment engine 140. For the purposes of the present disclosure, an "engine" may include one or more software modules, one or more hardware modules, or any combination thereof.

[0067]

[0076] The image sensor 106 can be used to generate image data 108 representing the attributes of the foods 102A - N, where N is any positive integer greater than 0 and represents the number of foods 102A - N on the conveyor belt 104. The image sensor 106 can be part of an imaging device such as the imaging device 160 described with reference to FIG. 1A. In the example of FIG. 1C, the image sensor 106 can be arranged in a manner that enables the image sensor 106 to capture image data 108 representing one or more images of the foods 102A - N as the foods 102A - N are advanced along the conveyor belt 104. In some implementations, the image sensor 106 can include one or more hyperspectral sensors configured to capture hyperspectral data representing the characteristics of the foods 102A - N. In such embodiments, each pixel of the hyperspectral image can correspond to the spectrum of infrared or ultraviolet light associated with the corresponding food imaged by a camera equipped with one or more sensors operating within the corresponding spectral range. The visible light spectrum can also be used to reconstruct the RGB images of the foods 102A - N described throughout this disclosure. In some implementations, the sensor 106 can be a low - resolution digital camera (e.g., 5M or less), a high - resolution digital camera (e.g., 5MP or more), or any other type of sensor capable of capturing the image data 108. The sensor 106 can cover a spectral range of 300 nm to 1700 nm. One or more other sensors, such as sensors within a hyperspectral camera, can cover wavelengths within the range of 300 nm to 2500 nm. One or more other cameras, including, but not limited to, the ultraviolet - capturing cameras described herein, can be used.

[0068]

[0077] In some implementations, sensor 106 may include multiple sensors positioned at multiple angles with respect to foods 102A - N. For example, sensor 106 may include a first camera that captures image data 108 of foods 102A - N from various projection angles and at least one additional second camera. In such a configuration, one or more additional cameras may be used to generate image data 108 based on wavelengths of light that are different from the wavelengths of light captured by the first camera or additional wavelengths of light. Generally, any combination of wavelengths of light may be acquired by sensor 106.

[0069]

[0078] Each particular camera of the one or more cameras may be configured to detect different or additional wavelengths of light in many different ways. For example, in some implementations, various sensors may be used within various cameras to detect various or additional wavelengths of light. Alternatively or in addition, each of the one or more cameras may be positioned at various heights, various angles, etc. relative to each other in an attempt to capture light of various wavelengths. In some implementations, the one or more cameras may be positioned to at least partially capture portions of foods 102A - N that may obstruct the field of view of the first camera.

[0070]

[0079] In some implementations, one or more light sources 105 can be used to illuminate the food items 102A - N such that the image sensor 106 can clearly capture the image data 108 of the food items 102A - N, as described with reference to FIG. 1A. The light source 105 can include one or more light sources each generating the same or different electromagnetic radiation. In this example, the light source 105 is depicted as being attached to the image sensor 106. In some implementations, the light source 105 can be positioned at one or more locations in the vicinity of the image sensor 106 to illuminate the food items 102A - N before and / or during the capture of the image data 108. In some implementations, the light source 105 can be selected based on the frequency of the electromagnetic radiation output. For example, in some implementations, the light source 105 can be a halogen light source. Alternatively or in addition, one or more light sources 105 can be a diode of a broadband light - emitting diode (LED), or a series of diodes, that can be used to provide light over the entire visible wavelength spectrum of light, the near - infrared wavelength spectrum, the electromagnetic spectrum, or any other spectrum.

[0071]

[0080] The light source 105 or the control unit of the light source 105 can be communicatively connected to the image sensor 106, or the control unit of the image sensor 106. For example, the image sensor 106 or the control unit of the image sensor 106 can send a signal to the light source 105 or the control unit of the light source 105 to cause the light source 105 to illuminate with one or more specific wavelengths of light at a specific output and / or at a specific instant. In some implementations, the specific instant can be a predetermined amount of time before or during the capture of the image data 108.

[0072]

[0081] The image data 108 generated by the image sensor 106 can be provided as an input to the extraction engine 110. The image data 108 can include one or more images of the foods 102A - N. Such images can also include one or more HSIs. In some implementations, the image sensor 106 can directly provide the image data 108 to the extraction engine 110. Next, the extraction engine 110 can process the image data 108. In some implementations, the image sensor 106 can store the image data 108 in a data store and / or a memory device. Next, the extraction engine 110 can access the data store and / or the memory device to obtain and process the image data 108.

[0073]

[0082] The extraction engine 110 can obtain / receive the image data 108. As shown in FIG. 1C, the image data 108 can be an image of one of the plurality of foods 102A - N. Thus, the image data 108 can be composed of a plurality of images 108A - N, where each of the images 108A - N corresponds to one of the foods 102A - N. The image data 108A - N can include depictions of the n foods 102A - N. The image data 108A - N can also include a part of the environment surrounding the foods 102A - N (such as the conveyor belt 104) or a part of the food processing facility (e.g., a pallet, flat, container, box, or other mechanism for holding the foods 102A - N). The extraction engine 110 can process the acquired image data 108 and extract a portion 112 of the image data 108, as described herein. As shown, the extracted portion 112 can include the foods 102A - N without the surrounding environment. In some implementations, the extracted portion 112 can include only one of the foods 102A - N.

[0074]

[0083] The extracted image portion 112 of the image data 108 (referred to as the extracted image 112 in this specification) can be provided as an input to the food identifier engine 120. In some implementations, the extraction engine 110 can directly provide the extracted image 112 to the food identifier engine 120. In some implementations, the extraction engine 110 can store the extracted image 112 in a memory device, and then the food identifier engine 120 can access the memory device.

[0075]

[0084] The food identifier engine 120 can use one or more object recognition algorithms / techniques to recognize portions of the extracted image 112 corresponding to the foods 102A - N. As a specific example, the foods 102A - N can be avocados. The food identifier engine 120 can be trained on multiple images of avocados to determine whether one or more avocados are depicted and which regions of the extracted image 112 contain avocados. Thus, the food identifier engine 120 can be trained to determine the bounding boxes around each of the foods 102A - N within the extracted image 112. The engine 120 can also be trained to generate output data in the form of annotated images 122 of the foods 102A - N. When generating the annotated image 122, the engine 120 can annotate or otherwise index each of the bounding boxes 122A - N representing each of the foods 102A - N. As described throughout this disclosure, the machine - learning trained model can then be applied to each of the bounding box regions 122A - N to determine quality metrics for each of the foods 102A - N.

[0076]

[0085] In some implementations, the annotated image 122 can include a coordinate system to annotate or index the location of each of the foods 102A - N. Numerical values such as the x and y values in an x,y coordinate system can be used to represent the location of the foods 102A - N within the annotated image 122. Subsequent processing steps can use the numerical values representing the location of the foods 102A - N to determine the outer boundaries of each of the foods 108A - N.

[0077]

[0086] The annotated image 122 generated by the food identifier engine 120 can be provided as an input to the quality assessment system 130. In some implementations, the food identifier engine 120 can directly provide the annotated image 122 to the quality assessment system 130. In other implementations, the food identifier engine 120 can store the annotated image in a memory device, and then the quality assessment engine 130 can access the memory device to obtain and process the annotated image 122.

[0078]

[0087] The quality evaluation engine 130 may be configured to determine quality metrics for each of the foods 108A - N depicted in the annotated image 122. As described throughout this disclosure, the quality evaluation engine 130 may use one or more machine - learning trained models to determine the quality metrics. Each of the models may be trained to identify various features indicative of various quality metrics for the foods 108A - N. For example, each model may be executed by various quality evaluation engines 132A - N. Each quality evaluation engine 132A - N may be configured to perform a specific quality evaluation operation on the annotated images 122A - N for each of the foods 108A - N. For example, one of the quality evaluation engines 132A - N may be configured to detect the presence of infection within the annotated images 122A - N of the foods 108A - N. Another of the quality evaluation engines 132A - N may be configured to detect a specific type of infection (such as latent infection or wound infection). Another of the quality evaluation engines 132A - N may be configured to detect the hardness of the foods 108A - N within the annotated images 122A - N. Yet another of the quality evaluation engines 132A - N may be configured to detect the ripeness of the foods 108A - N within the annotated images 122A - N. In some implementations, one or more of the engines 132A - N may be executed in series. Sometimes, one or more of the engines 132A - N may be executed in parallel. Executing in parallel may be advantageous for reducing the amount of time required to process the annotated image 122 and for performing multiple quality evaluations of the foods depicted therein. In some implementations, only some of the engines 132A - N may be selected for execution either in series or in parallel.

[0079]

[0088] In some implementations, a set of quality evaluation engines 132A - N may be configured based on the type of foods 108A - N being analyzed. The quality evaluation engines 132A - N may also be configured based on the business practices of the entity implementing the system 100 or the stakeholders throughout the supply chain of the foods 108A - N.

[0080]

[0089] Still referring to FIG. 1C, the quality assessment engine 140 can be configured to evaluate the overall quality of the foods 108A - N based on the quality metrics determined by the engines 132A - N. For example, the engine 140 can determine the aggregate quality of batches of the foods 108A - N. The engine 140 can also determine, for each of the foods 108A - N, the percentage of the surface area of the food covered by the detected infection. In some implementations, the engine 140 can also determine the current edibility (e.g., consumption rating) of the foods 108A - N and / or the length of time of edibility of the foods 108A - N (e.g., based on how much infection has been detected within a particular food or within a batch of foods and / or based on the percentage of coverage of infection on a particular food or batch of foods).

[0081]

[0090] As described herein, the engine 140 can also determine one or more supply chain modifications based on the quality metrics of the foods 108A - N. For example, if the engine 132A determines that the foods 108A - N are infected, and the engine 140 determines that approximately 80 - 90% of the surface area of each of the foods 108A - N (or the majority of the foods 108A - N) within the batch is covered by the infection, the engine 140 can further trigger an update to the distribution plan of the foods 108A - N so that they are delivered to the food processing plant so as not to be wasted. If the engine 140 determines that these foods 108A - N still have a certain shelf life and are thus still consumable for a period of time, the engine 140 can generate instructions for delivering the foods 108A - N to the nearest retail environment so that they can be purchased and consumed by the end user before the shelf life / edibility expires. Innumerable other exemplary operations can be determined and performed based on the quality metrics determined by the engines 132A - N (e.g., the engine can provide a quantitative means for differentiating between various batches of foods based on an aggregate quality score).

[0082]

[0091] As described throughout this disclosure, the quality metrics generated by engines 132A - N can be numerical, binary, and / or boolean values. The quality metric can be a numerical value on a given scale. The quality metric can also be values such as "good", "bad", "inferior", "ok", "excellent", "moderate", "sufficient", etc. As another example, the output data can be a vector of one of two different values for each of the depicted foods 108A - N (providing an indicator of quality pass or quality fail (e.g., healthy or unhealthy / infected) for each food 108A - N). The overall quality metric can also be determined for each of the depicted foods 108A - N based on a vector of quality metrics output by engines 132A - N. One or more different values can be defined based on the supply chain and / or the practices of relevant stakeholders within the supply chain.

[0083]

[0092] In some implementations, overall quality scores 134A - N can be generated for each of the foods 108A - N in the output image 134. The overall quality scores 134A - N can be numerical indicators of how much infection is present in a particular food 108A - N and / or what probability a particular food 108A - N is infected. The higher the score on the numerical scale, the higher the probability of infection present and / or the higher the probability of being present within the particular food. The lower the score on the numerical scale, the lower the probability of infection present and / or the lower the probability of being present within the particular food. The numerical scale shown in FIG. 1C can be on a scale of 0 - 1. One or more other numerical scales (including but not limited to 0 - 5, 0 - 3, 1 - 5, 1 - 3, 1 - 4, 0 - 100, etc.) can be used.

[0084]

[0093] The overall quality scores 134A - N can be examined by relevant stakeholders within the supply chain. The stakeholders can examine the scores 134A - N and determine one or more modifications to the supply chain. The quality metrics generated by engines 132A - N can also be output in one or more other formats for scrutiny by relevant stakeholders.

[0085]

[0094] FIGS. 2A-B are flowcharts of a process 200 for determining food quality based on image data. Process 200 can be performed by computer system 150. Process 200 can also be performed by one or more other computing systems, devices, computers, networks, cloud-based systems, and / or cloud-based services. For illustrative purposes, process 200 is described from the perspective of a computer system.

[0086]

[0095] Referring to process 200 in both FIGS. 2A-B, the computer system can receive training image data at block 202. As consistently described, the image data can be captured under UV illumination conditions. Some of the image training data can be captured under other illumination conditions (such as in the visible light spectrum and / or short-wave visible light). Capturing image training data under other illumination conditions may require the use of at least one narrowband pass filter between the food being imaged and the imaging device so that the fluorescence signal from the food is not masked by visible light from the light source. The training image data can be received from an imaging device as described in FIG. 1A. The training image data can also be retrieved from a database, data store, or other repository that stores image data of a particular food. The image data can include images of a particular food whose quality is considered to be determined based on various features that are visible or non-visible from the image data. The training image data can be captured from the food at one or more different times throughout the food supply chain. For example, the training image data can be captured from the food when it arrives at a storage facility, during transport within the storage facility to a storage location, when it is transported from the storage facility to the final consumer retail environment, when it arrives at the final consumer retail environment, etc., before being transported from the farm to the storage facility.

[0087]

[0096] In block 204, the computer system can identify foods within the training image data. The computer system can use object detection techniques to generate a bounding box around each food within the training image data. As described herein, the model can be trained with high precision by using one food within the training image data. The model can also be trained, validated, and / or tested by using multiple foods within the training image data. The computer system can also generate a numerical grid index for the bounding box such that each grid index represents a different food within the training image data. The grid index can be used to identify foods in a data store or other storage technique.

[0088]

[0097] Optionally, as part of identifying features, the computer system can attach a ground truth label to the image data. Features such as spoilage, mold, ripeness, dryness, damage, etc. can be identified from the training image data and accordingly labeled / annotated for each food within the image data. In some implementations, one or more features such as hardness and dry matter content may not be visible from the training image data and thus can be labeled by using other techniques (such as destructive techniques involving a durometer and / or penetrometer that puncture or penetrate the skin, surface, peel, or flesh of the foods within the training image data).

[0089]

[0098] The computer system may also select food at block 206. The computer system may use a random number generator to randomly select a bounding box from the training image data. Next, the food within the selected box can be used for training purposes. In some implementations, the computer system may select a bounding box that includes food having a threshold amount of feature labels. The computer system may also select a bounding box having food that meets other threshold feature criteria. The threshold feature criteria may correspond to features indicating infection within a particular type of food. For example, the computer system may select a bounding box that includes food having modeled features (including bruises, rot, mold, drying, browning, other discoloration of the food's peel / surface / skin, etc.).

[0090]

[0099] The computer system may, at block 208, identify and cluster the food features of the selected food. As described herein, the computer system may analyze the pixel data within the bounding box of the selected food to differentiate the pixels and categorize them into clusters. The first cluster may correspond to pixels representing the background within the bounding box. The second cluster may correspond to pixels representing healthy portions of the food (e.g., a portion of the surface / peel / skin of the food that is not labeled as indicating infection). The third cluster may correspond to pixels representing unhealthy portions of the food (e.g., a portion of the surface / peel / skin of the food that includes features such as bruises, mold, rot, drying, discoloration, etc. that indicate infection). At block 208, additional or fewer clusters may also be generated. For example, the third cluster may be further decomposed into additional sub - clusters to provide a more granular analysis of the food's quality. The sub - clusters may, by way of example, correspond to pixels representing the center / source of mold on the food. Another sub - cluster may correspond to pixels representing the spread of mold from the center / source across the surface of the food. As another example, the sub - clusters may correspond to different degrees of mold. One exemplary sub - cluster may be expressed white mold, another sub - cluster may correspond to blue mold, and another sub - cluster may correspond to a latent infection that is not visible under visible light conditions but is present only under UV light conditions.

[0091]

[0100] Optionally, in block 210, the computer system may map the clusters into a multi-color space. The multi-color space may be an RGB space. Mapping the clusters into a multi-color space may provide a clearer visualization of the various features represented in the training image data. Mapping the clusters into a multi-color space may also provide a clearer visualization of infection features or other unhealthy features in the food (which cause the food to have a lower quality at the current time and / or over a period of time). Such features may otherwise not be easily visible to the human eye or within the visible light spectrum.

[0092]

[0101] The computer system may train the model to detect food characteristics at block 212. The computer system may train the model by using the optionally mapped clusters from block 210. Model training may be performed by using the k-means clustering technique. One or more other clustering techniques may also be used for model training. For example, Gaussian Mixture Model (GMM), K-nearest neighbor (KNN), K-median, hierarchical clustering, and / or Density-Based Spatial Clustering of Applications with Noise (DBSCAN) techniques may be used for model training. In some other implementations where the pixels are not easily clustered within the healthy / infected clusters, a supervised instance segmentation model (e.g., CNN-based) may be trained. In this example, a training dataset with labels may be generated, and various sections of the food may be manually annotated. Next, the model is applied to each bounding box in the image data of the food, and a segmentation map (e.g., a segmented image) may be predicted. Next, the segmentation map may be used instead of (or in addition to) the cluster map. The segmentation map may be used in the same way that the clusters may be used in the disclosed techniques of process 200.

[0093]

[0102] The computer system can also train two or more models at block 212. The computer system can train models for each food type. The computer system can also train models for each quality characteristic of the food (e.g., ripeness, hardness, infection, potential infection, wound infection, bruising, spoilage, mold, dryness, sourness, taste, etc.). The trained models can be stored in a data store for retrieval during runtime use. The models can also be stored in local memory within the computer system during runtime use. Refer to FIG. 1B for additional discussion regarding training the models.

[0094]

[0103] The models generated at block 212 can be trained to receive, as input, image data of a food (or batch of foods). By using the models, the computer system can identify features indicating infection within the food. Next, the models can return an indicator of the presence of infection within the food as output. The model output can be a numerical value, a boolean value, and / or a string value.

[0095]

[0104] The numerical value can be, for example, a value on a predetermined scale. The predetermined scale can be, for example, 0 to 3.5. On this scale, a value close to 0 can indicate the absence or slight presence of infection within the food. A value close to 3.5 can indicate a certain amount of infection present within the food. As another example, a value close to 0 can indicate a slight likelihood or no likelihood of the presence of infection, while a value close to 3.5 can indicate a higher likelihood of the presence of infection within the food.

[0096]

[0105] The boolean value can be a binary value such as true / false, Yes / No, 0 / 1, infected / uninfected, etc. The boolean value can indicate whether infection is present within the food. The string value can indicate whether infection is present and / or the likelihood of the presence of infection. For example, the string value output by the model can indicate "infection is present within the food", "infection is not present within the food", "infection is likely present", and / or the degree of the presence of infection within the food. One or more other outputs can be generated by the model.

[0097]

[0106] During runtime, the computer system may receive image data of food (block 214). As described with reference to FIGS. 1A and 1C, the image data may be captured under UV light conditions and received from an imaging device such as imaging device 160 and / or image sensor 106. The food in the image data may be of the same type as the food in the training image data. In some implementations, the image data received at block 214 may be of just one food item. The image data may also include a batch, flat, pallet, batch or group of foods of the same food type.

[0098]

[0107] The computer system may apply the model to the image data at block 216 to determine the presence of infection within the food. The computer system may determine the presence of infection within each food in the image data. For example, the computer system may perform object detection techniques to identify a bounding box around each food in the image data. Next, the model may receive each bounding box as an input for determining the presence of infection of a particular food within the bounding box. The computer system may also determine the overall presence of infection of the foods in the image data. For example, the model may receive the image data as an input and determine the average presence of infection of the foods. The model may also determine the presence of infection of each food in the image data. Next, the computer system may average, sum, or otherwise aggregate the presence of infection for each food to determine the overall presence of infection of the batch of foods represented in the image data.

[0099]

[0108] As described herein, the model can be used to analyze color features within the image data. The colors within the image data can be clustered and mapped within a color space (e.g., RGB space, CIELAB space, or other three-dimensional color space). Next, the model can analyze each of the clusters within the color space to determine whether an infection is present. For example, the computer system can determine that "an infection is present if the cluster associated with the infection has a pixel size and / or color value that exceeds a threshold infection determination criterion." The computer system can also determine that an infection is present if the cluster associated with the infection is located within an area of the image data other than the border or edge of the image data (which can indicate the background of the image data). The threshold infection determination criterion can be based on historical data and / or other factors such as customer preferences, the quality of the food that the customer will still purchase and / or consume, and / or the level of infection presence. This process can be effective for accurately evaluating and scoring the infection characteristics of food even if they are not easily apparent within the visible light spectrum. This process can also be computationally easy and fast, thereby utilizing less time and computational resources to determine the quality of food based on image data captured under UV light conditions.

[0100]

[0109] In some implementations, the computer system may select which model or models to apply to the image data based on the type of food being imaged. For example, the computer system may use object recognition techniques to identify the type of food in the image data. The computer system may also identify the food type by applying an identifier model to the image data. The model may be trained by using machine learning techniques such as deep neural networks, CNNs, etc. to detect the food in the image data and then identify the type of food. Next, the computer system may retrieve one or more models associated with the identified type of food from a data store or other database that stores the generated models. In some implementations, the computer system may use metadata, which is a part of the image data, to determine which models to retrieve. The metadata may indicate, for example, the country of origin of the food, the geographical location of the country of origin, the food type, the growing conditions, the known / historical ripening conditions, etc.

[0101]

[0110] In block 218, the computer system may determine whether an infection has been detected in the food based on the model output. As described above, the model may be trained to generate an output indicating the presence of an infection in the food. In block 218, the computer system may return the output. The computer system may also normalize, process, or otherwise transform the model output into an indicator of the presence of an infection for a particular food and / or batch of foods in the image data.

[0102]

[0111] If no infection is detected, the computer system may return an indicator of healthy food (block 220). For example, the computer system may determine that no infection is present in the food if the model output is below a predetermined threshold infection level indicating the presence of an infection. The computer system may also determine that no infection is present in the food if the model output indicates that no infection is present in the food. In some implementations, if any amount of infection is detected in the food, the computer system may determine that an infection is present.

[0103]

[0112] Returning an indicator of a healthy food may include storing the indicator in a data store for later retrieval, use, and / or output. Returning an indicator may also include sending the indicator to a user device of a stakeholder within the food supply chain. The indicator may then be used by a computer system and / or stakeholder to determine, generate, and / or implement one or more supply chain modifications. See FIGS. 3A - B for additional discussion regarding supply chain modifications.

[0104]

[0113] If an infection is detected, the computer system may determine, at block 222, the percentage of food infection coverage. The computer system may make this determination for each food identified as having an infection present. To determine the percentage of coverage, the computer system may sum the amount of pixels indicating the infected portion of the food and divide this sum by all the pixels indicating the infected and healthy portions of the food. The computer system may then multiply the result of this division by 100 to obtain a percentage value indicating what amount of the total surface of the food is infected.

[0105]

[0114] In some implementations, the model may be trained to determine, at block 212, the percentage of infection coverage within a food determined to have an infection present. Further, in some implementations, since the model may be trained to directly predict the percentage of infection coverage from the image data, blocks 208 and 210 may not be performed. Sometimes, the model may directly predict various segments such as healthy segments, infected segments, and background segments within the image data and then use those segments to calculate the percentage of infection coverage. In some implementations, the computer system may apply a model generated and trained to determine the percentage of infection coverage of an infected food at block 222.

[0106]

[0115] The computer system can return an indicator of the presence of infection and the percentage of infection coverage at block 224. The computer system can return an indicator for each food item within the image data. The computer system can also return an aggregated value for a batch of food (e.g., by summing, averaging, and / or aggregating the indicators of infected food relative to the total amount of food within the image data). The indicator of the presence of infection and the percentage of infection coverage can be represented by the image data. For example, a bounding box of the infected food where the food is shown within a multi-color space (e.g., an RGB space) can be returned. In a multi-color space, the infected portion of the food can be clearly visualized not only from the healthy portion of the food but also from the background within the bounding box. The bounding box can also include an indicator of the percentage of infection coverage for a particular food item.

[0107]

[0116] As described above with respect to block 220, returning an indicator at block 224 can include storing the indicator within a data store. Returning an indicator at block 224 can also include transmitting the indicator to a stakeholder's user device for output within a graphical user interface (GUI) display. Next, the indicator can be used by the computer system and / or stakeholders to determine one or more supply chain modifications for a particular food item and / or batch of food. See FIGS. 3A - B for additional discussion regarding supply chain modifications.

[0108]

[0117] Optionally, the computer system can determine and return the length of time of food edibility at block 226. Given a sufficient amount of time series data (e.g., 4 weeks) of the progression of infection on food over the long term, combined with metadata such as country of origin, harvest date, storage temperature, etc., the model can be constructed to predict the growth of infection on food over the long term and thus estimate when the infection is about to reach the edibility threshold. As a specific example, the food can be evaluated for fluorescence at time = 0. The fluorescence can be categorized based on the fluorescence signal (e.g., no fluorescence, low fluorescence, medium fluorescence, high fluorescence). The food can be cultured under relevant conditions and tracked for damage over a predetermined amount of time (such as the next 4 weeks). Next, the amount of damage per image over the predetermined amount of time can be calculated. When new food is analyzed, the amount of fluorescence detected for these new foods can be correlated / associated with and returned to a model constructed from the original data captured over the above-mentioned 4 weeks. As a result, the computer system can estimate the risk of damage in new food at a future time point.

[0109]

[0118] The determination in block 226 can be made for each food in the image data, for each food in the image data having an infection, and / or as an aggregated value for the entire batch of foods in the image data. The computer system can apply the edibility model to the image data of the food identified as being infected. The edibility model can be trained to predict the length of time of edibility (e.g., shelf life, consumability, ripeness, etc.) based on the presence of an infection, the type of infection, the spread of the infection, the percentage of infection coverage, and other information regarding a particular type of food (e.g., origin, transportation conditions, historical edibility conditions, historical ripening conditions, etc.). The length of time of edibility can indicate how long the food is in good condition for consumption by the end consumer. Sometimes, a food having an infection present can still be edible / good for consumption for a period following the start time of the infection. The predicted length of time of edibility can be used by the computer system and / or stakeholders to determine supply chain modifications to reduce potential waste of the food. For additional discussion regarding the determination of the length of time of edibility of a food, see FIG. 4.

[0110]

[0119] Blocks 202-212 can be performed at a different time than blocks 214-226 (e.g., some amount of time elapses between blocks 202-212 and blocks 214-226). For example, blocks 202-212 can be performed during a training phase. Blocks 214-226 can be performed at a later time as part of the runtime use of the model generated and trained during the training phase. In some implementations, blocks 202-212 can be performed immediately following blocks 214-226.

[0111]

[0120] FIGS. 3A - B are flowcharts of a process 300 for determining the quality of food based on image data. Process 300 can be performed by computer system 150. Process 300 can also be performed by one or more other computing systems, devices, computers, networks, cloud - based systems, and / or cloud - based services. For illustrative purposes, process 300 is described from the perspective of a computer system.

[0112]

[0121] Referring to process 300 in both FIGS. 3A - B, at block 302, the computer system can receive image data of food. See FIGS. 1A, 1C and FIGS. 2A - B for further discussion.

[0113]

[0122] At block 304, the computer system can perform object detection techniques to identify the bounding boxes around each food item within the image data. The computer system can utilize one or more machine - learning models that are trained to identify food items within a food analyzer and / or within the bounding boxes. A CNN and / or an image classification model can be used to actively identify the food items within the bounding boxes. Each of the returned bounding boxes can be processed (e.g., serially) in a separate job. In some implementations, the returned bounding boxes can also be processed in parallel. In some implementations, the computer system can pre - identify food items by analyzing the metadata associated with the image data. Sometimes, for example, the metadata can indicate what types of food items appear within the image data. Next, the food type can be used by the computer system to search for a food analyzer or other image classification model that is used to detect and identify each of the food types within the image data. Further, if no food items appear within the image data, the bounding boxes can be returned for further processing and analysis.

[0114]

[0123] The computer system can determine a lattice structure based on the bounding boxes at block 306. The computer system can perform indexing and can assign these indices to each of the bounding boxes that make up the lattice structure of the image data. Each of the identified foods can receive one of the indices that can be used to identify the food. To determine the lattice structure, the computer system can (1) find what appears most likely to represent a column of foods based on the Y height of the bounding box, and then (2) sort the entire data frame based on the X position. A lattice can be determined, and each bounding box within the lattice structure can be assigned an index value that can be used to identify the foods that appear within the bounding box. One or more machine learning trained models can also be used to determine the lattice structure and assign indices.

[0115]

[0124] Determining the lattice structure and indexing it can be advantageous for associating the determined quality metric with the foods that appear in the image data. After all, each food can have different quality metrics (e.g., health, infection, mold, spoilage, etc.), so each determined quality metric should be assigned to the index of the corresponding food. Additionally, assigning index values to foods can be advantageous for correlating additional metrics and data regarding a particular food with the quality metric determined by the computer system. As a result, the computer system can construct a more robust and accurate quality metric for a particular food within the image data.

[0116]

[0125] In some implementations, the grid structure may also be advantageous for facilitating the exploration of all image data of food to identify and / or output food having certain characteristics and / or quality metrics (e.g., outputting all food having a percentage infection coverage exceeding a certain threshold infection level). This may be beneficial for training the models described herein to more accurately identify and judge the quality characteristics of food. This may also be beneficial for stakeholders within the supply chain who are interested in monitoring food, analyzing ROIs, and / or performing supply chain adjustments that help reduce or otherwise prevent food-based waste.

[0117]

[0126] The computer system may select a bounding box of the food within the image data (block 308). The computer system may select one of the foods depicted within the image data for evaluation by using the models described herein.

[0118]

[0127] Next, at block 310, the computer system may apply a clustering model, such as a k-means model, to the selected bounding box. The computer system may pass the bounding box portion of the image data through a k-means model that evaluates the quality of the food based on pixel data (e.g., color). The model is described as a k-means model, but the model may also be any other model trained by clustering techniques as described herein. The model may return an indicator of whether there is an infection within the food. The computer system may also select one or more additional / other models for application to the bounding box. The models may be selected based on any characteristics that may be desirable for the evaluation of the type of food, user (e.g., stakeholder) preferences, and / or quality of the food (e.g., ripeness, hardness, shelf life, edibility, length of time of edibility, etc.). Each model may be executed independently of the others. The models may be executed serially. In some implementations, the models may be executed in parallel.

[0119]

[0128] Executing multiple models in block 310 can be advantageous for generating a robust and accurate quality assessment of food. As a specific example, the first model can identify browning on the surface of the fruit. The first model can extract features within the image data indicating browning. These extracted features, which can be trained to identify the ripeness stage of the fruit based on the browning further, can be provided as an input to the second model. The determination of the second model of the ripeness stage of the fruit can be provided as an input to the third model. The third model can be trained to determine an overall food quality metric score based on the ripeness stage based on the browning further. Thus, a more robust and less invasive quality assessment can be performed. It can be understood that any number of models can be used to perform the quality assessment described herein in any order.

[0120]

[0129] The computer system can determine a food quality metric based on the output from the clustering model (block 312). As described with reference to FIGS. 1-2, the computer system can determine whether an infection is present in the food. The computer system can determine the percentage of the food's infection coverage. The computer system can also determine the current edibility of the food and / or the length of time of the food's edibility (e.g., remaining shelf life for consumption by the end user, remaining time, etc.).

[0121]

[0130] The computer system may also store the food quality metric in block 314 by using the food boundary box grid index. The computer system may associate the quality metric with the grid index assigned to the food. The quality metric may then be stored in a data store or in other types of databases having this association. The quality metric may be retrieved by the computer system and presented on one or more user devices as described herein. The quality metric may be used during future analysis and monitoring of a particular food (or batch of foods “of which / associated with” the particular food is a part). In some implementations, the quality metric may also be used in future training data sets to refine and improve the accuracy of the k-means model or other models described herein.

[0122]

[0131] In block 316, the computer system may determine whether there is additional food within the image data. For example, the computer system may determine whether there are labeled boundary boxes within the grid structure for which a quality metric has not yet been assigned. In some implementations, the computer system may query the data store in block 316 to make a determination by looking at which grid indices have had a quality metric assigned.

[0123]

[0132] If there is additional food, the computer system may return to block 308 and repeat blocks 308 - 314 for each remaining food within the image data. If there is no longer any food within the image data, the computer system may proceed to block 318.

[0124]

[0133] In block 318, the computer system can output quality metrics of food within the image data. The quality metrics can be presented in many ways. For example, the quality metrics can be depicted by using an image representing the food or a portion of the image data. Food classified as infected can be represented by an output that includes a close-up image of a wound or other feature on the food that makes the food appear of poor quality and thus infected. As another example, the quality metrics can be depicted by using a spectrogram. The quality metrics can also be output as numerical values, boolean values, and / or strings as described herein.

[0125]

[0134] In some implementations, the computer system can output one or more quality metrics for one or more foods based on user preferences. For example, a user on a user device can provide an input to the user device that requests to examine the quality metrics of a subset of the foods depicted within the image data. This input can also request to examine a subset of the quality metrics of a specific food represented within the image data. One or more other user inputs can be used to generate a custom / personalized quality metric output for display on the user device.

[0126]

[0135] Optionally, the computer system can determine and return one or more supply chain modifications based on the quality metrics of the food within the image data (block 320). The modifications can include changing the location where the food is shipped, changing the amount of time the food is stored, changing the storage conditions, applying a ripening agent or other treatment to the food, discarding the food, moving the food for shipment to the end consumer, moving the food for shipment to a food processing factory, and so on. The supply chain modifications can vary depending on the quality metrics of the food.

[0127]

[0136] For example, if the food is classified as being infected and / or if the percentage of infection coverage exceeds a threshold infection level, the computer system may determine that the food should be shipped to a food processing factory and / or should be delivered to the grocery store geographically closest to the current location of the food. Such supply chain modifications can reduce the likelihood that the food will be wasted. Similarly, if the predicted remaining time of edibility of the food is shorter than a threshold length of time, the computer system may generate an instruction to move the food and / or a batch of food containing the food to the final consumer retail environment geographically closest to the current location of the food so that the food can be purchased and consumed by the final consumer before it becomes no longer edible / consumable.

[0128]

[0137] As another example, if the food is scored as being healthy (e.g., good quality, free of the presence of infection, infected but the infection is below a threshold infection level, etc.), the computer system may determine that the food can be stored for a longer period of time than other foods and / or that the food can be shipped to a grocery store that is geographically far from the current location of the food. One or more other modifications may be possible based on what quality is determined for the food. One or more other modifications may also be possible based on user-specified preferences associated with various quality metrics evaluated by the models described herein.

[0129]

[0138] FIG. 4 is a flowchart of a process 400 for determining the length of time of edibility of food based on their determined quality. The food may still be edible when there is a small presence of infection (e.g., less than a threshold amount of yellow UV fluorescence is identified within the image data of the food). Accordingly, process 400 may be used to predict how long the food will last for purchase and consumption from the time the infection is detected.

[0130]

[0139] Process 400 can be performed by computer system 150. Process 400 can also be performed by one or more other computing systems, devices, computers, networks, cloud-based systems, and / or cloud-based services. For illustrative purposes, Process 400 is described from the perspective of a computer system.

[0131]

[0140] Referring to Process 400 of FIG. 4, the computer system may receive, at block 402, time-series training image data of a large amount of food identified as being infected. The image data may be captured at various points in time or over a predetermined period within the life cycle or supply chain of the large amount of food. The image data may be captured daily (e.g., once a day) throughout the life cycle of the large amount of food. The image data may also be captured at other predetermined time intervals during the life cycles of multiple foods. The image data may be captured under UV light settings. In some implementations, the image data may also be captured under other light settings, such as under visible light conditions.

[0132]

[0141] The computer system may train an edibility model (block 404) to predict the length of time of food edibility based on the time-series training image data. The time-series training image data may demonstrate / track the progression and / or growth of infection within the food over a long period. The time-series image data may be labeled at various stages of infection progression and / or growth. The various stages of infection progression and / or growth may be correlated with other training data indicating the edibility of the food (e.g., shelf life, freshness, ripeness, ease of consumption, etc.). The other training data may be obtained by using destructive techniques such as incisions, punctures, or insertions into the food. The other training data may also be obtained by using destructive techniques with a penetrometer and / or durometer. Further, other training data indicating the edibility of the food may include the taste, freshness, ripeness, hardness, etc. of the food during its life cycle.

[0133]

[0142] Next, the edibility model can be trained to predict a food edibility metric by using labeled time series training image data and other training data indicating edibility. The model can be trained by using machine learning techniques such as CNN and / or deep neural networks. The edibility model can also be trained to identify colors in the image data indicating infection in the food and then correlate those colors with various levels of edibility and / or length of time of edibility. For example, image data having at least a threshold amount of yellow UV fluorescence can represent a food having a shorter remaining amount of edibility time than image data having an amount of yellow UV fluorescence less than the threshold amount. Yellow UV fluorescence can be correlated with infection in the food. The threshold amount of yellow UV fluorescence in the image data can also vary depending on the type of food in the image data.

[0134]

[0143] The edibility model can generate an output indicating the edibility level of the food. In some implementations, the model can also generate an output indicating the length of time of edibility of the food. The output can be a numerical value, a boolean value, and / or a string value. For example, the model can generate a score indicating how edible the food is. The score can be assigned a numerical value on a predetermined scale (e.g., a value on a scale of 1 to 100). A value less than a predetermined threshold (e.g., a value of 1 to 50) can indicate that the food is not very edible or has a shorter remaining edibility length. A value exceeding the predetermined threshold (e.g., a value of 51 to 100) can indicate that the food is edible or has a longer remaining amount of time of edibility. The model can also generate an output indicating the predicted amount of time of the remaining edibility of the food. For example, the output can include the number of hours, days, weeks, etc. that the food is predicted to still be edible and consumable by the end consumer.

[0135]

[0144] Once the edibility model is generated and trained, the model can be stored in a data store for later time search for runtime use. The edibility model can also be stored in local memory for runtime use in a computer system.

[0136]

[0145] During runtime, the computer system can receive food image data at block 406. The food can be of the same type as the food in the time series training image data. At block 406, the computer system can receive only the image data of the food identified as having an infection, as described in FIGS. 1-3. The image data can be of one food. The image data can also be of a batch of foods. In some implementations, the computer system can receive the image data of the food regardless of whether it is identified as having an infection. Thus, process 400 can be performed to determine the length of time of edibility of not only healthy foods but also infected foods or foods having other poor quality characteristics.

[0137]

[0146] The computer system can apply the edibility model to the image data to determine food edibility at block 408. The image data can be provided as an input to the edibility model. The edibility model can generate an output indicating whether the food in the image data is edible and / or a predicted length of time of the edibility of the food. Thus, as part of determining food edibility, the computer system can predict the length of time of edibility based on food edibility (block 410). As described above, the prediction can be made by using the edibility model. The length of time of edibility can indicate the amount of remaining time that the food (or batch of foods) in the image data can be consumed by the end consumer.

[0138]

[0147] The computer system can, at block 412, restore the length of the time of edibility of the food. The computer system can restore the length of the time of edibility of the contaminated food in a batch of food (e.g., food having a quality metric that meets a threshold contamination determination criterion such as the most contaminated food or percentage of contamination coverage). The computer system can also restore the total length / average length of the remaining time of edibility of all the food in the batch. The computer system can also restore the length of the time of edibility of each contaminated food in the image data.

[0139]

[0148] Optionally, the computer system can determine a supply chain modification (block 414) based on the length of the time of edibility of the food or batch of food. For additional discussion regarding determining a supply chain modification, see process 300 of FIGS. 3A - B.

[0140]

[0149] FIG. 5A depicts an exemplary contamination detection analysis 500 using image data of food 501 under UV illumination. In this example, food 501 is a mandarin. One or more other foods such as other citrus fruits can also be imaged as described throughout the present disclosure. By using the disclosed technology, UV light image data 502 can be captured by an imaging device of food 501. The imaging device can capture an image of a batch, pallet, or flat of similar food 501. Image data 502 can be a bounding box cropped from the batch of image data, where image data 502 depicts one of the batch of food 501. As shown within image data 502, the UV light conditions cause fluorescence in the contaminated portions 503A and 503N of food 501. Contaminated portion 503A fluoresces (e.g., yellow) within image data 502 while contaminated portion 503N represents mold (e.g., white) on the surface of food 501. In image data captured under normal visible light settings, contaminated portions 503A and / or 503N may not be as visible or distinct. Instead, contaminated portion 503A may be visible, for example, but the spread of contaminated portion 503A across the surface of food 501 may not be visibly distinct.

[0141]

[0150] Image data 502 can be processed by a computer system (e.g., computer system 150) by applying a model (e.g., a k-means clustering model, another model using a clustering algorithm (such as a Gaussian mixture model (GMM))) to identify clusters of pixels representing various features within the image data 502. The model output can include processed image data 504, where the features of the image data 502 are clustered and mapped within the RGB space. Each cluster within the processed image data 504 can be indicated by a color representing different features within the image data 502.

[0142]

[0151] For example, a cluster of pixels 508 can represent the background of the image data 502. A cluster of pixels 510 can represent the healthy features of the food 501 from the image data 502. Clusters of pixels 512A and 512N can represent the unhealthy or infected features of the food 501 from the image data 502 (such as the infected portions 503A and 503N of the food 501, respectively). In this specific example, cluster 508 can be green, cluster 510 can be purple, and cluster 512A can be yellow, and cluster 512N can be blue. In some implementations, one or more other different colors can be used to represent the clusters 508, 510, 512A, and / or 512N.

[0143]

[0152] The processed image data 504 as shown in FIG. 5A provides a clear visualization of the infection within the food 501. The computer system described herein may also generate an infection coverage metric 506. The infection coverage metric 506 may indicate how much of the food 501 that is visible within the image data 502 is covered by the infection. Each side of the food 501 may be imaged as described herein to determine the overall infection coverage of the entire surface of the food 501. In some implementations, the computer system described herein may also estimate / measure the infection on the other sides of the food 501 that are not visually depicted within the image data 502. For example, the computer system may estimate that the other side of the food 501 (the second half of the surface of the food not shown within the image data) is healthy and thus calculate the infection coverage of the entire surface of the food 501.

[0144]

[0153] The infection coverage metric 506 may be output or returned along with the processed image data 504 for presentation within the GUI display on the user device described herein. The infection coverage metric 506 may be the average of the pixels within the infection clusters 512A - N for the pixels within both the healthy clusters 510 and the infection clusters 512A - N shown within the processed image data 504. In the example of FIG. 5A, the food 501 has an infection coverage metric 506 of 0.182. This value may correspond to 18.2% of the infection coverage when multiplied by 100 to generate a percentage value. The infection coverage metric 506 may also be presented as a percentage value as described throughout this disclosure.

[0145]

[0154] The infection coverage metric 506 indicates how much of the surface of the food 501 is infected. Typically, metric 506 can be used to infer that the interior of the food 501 may also be infected and that more of the interior of the food 501 may be infected than the outer surface of the food 501. Some infections, such as by a pathogen (e.g., Penicillium sp.), may remain near the surface of the food 501 and thus may not infect much of the interior of the food 501. Thus, the bruise type indicia in the image data typically indicate surface damage to the food 501. On the other hand, potential infections typically pass through the interior of the food 501 and thus spread beneath the surface of the food 501. Thus, the potential indicia in the image data typically indicate that the food 501 may also include some level of internal damage.

[0146]

[0155] FIG. 5B depicts an exemplary infection detection analysis 550 that uses image data of a batch of food under UV illumination. Here, the image data 552 is captured from the oranges of the batch under UV light conditions. As shown, various colors fluoresce at each of the oranges, indicating that each of the oranges may have some amount of infection present. The fluorescent colors can indicate the center point of the infection and the spread of the infection across the entire surface of each orange. By processing the image data 552 using the techniques described herein, the image data 554 can be arrived at.

[0147]

[0156] Processing the image data 552 involves applying the k-means clustering model or another model that implements the clustering technique described herein. The model can generate an output (such as the processed image data 554), where clusters of features (e.g., background, healthy part of the food, infected part of the food) are mapped within the RGB space. Each cluster within the processed image data 554 can be presented in a different color. The computer system described herein can determine how many of the foods within the processed image data 554 have a color representing infection. The computer system can also determine the percentage of the infection coverage for each food within the processed image data 554. Further, the computer system can determine or predict the length of the edibility of each food within the processed image data 554 and / or the length of the edibility of the entire batch of foods within the processed image data 554.

[0148]

[0157] FIG. 6 is an exemplary block diagram of components used to determine food quality based on the techniques described herein. The computer system 150, the food quality data store 620, and the model data store 618 can be in communication (e.g., wired and / or wireless) via the network 180.

[0149]

[0158] Computer system 150 may include an object detection engine 602 (e.g., the food identifier engine 120 of FIG. 1C, the extraction engine 110 of FIG. 1C, and / or a combination of the food identifier engine 120 and the extraction engine 110), an indexing engine 604, a quality assessment engine 606 (e.g., the quality assessment system 130 of FIG. 1C, the quality assessment engine 140 or a combination thereof), a supply chain modifier 607, a model training engine 608, and a communication interface 610. The communication interface 610 may provide communication between the components described herein. In some implementations, one or more of the components 602, 604, 606, 607, 608, and 610 may be separate from the computer system 150 and from one or more other computer systems, computers, servers, devices, and / or parts of a network.

[0150]

[0159] The model training engine 608 may be configured to generate one or more models that can be used by the quality assessment engine 606. The engine 608 may perform the training described herein (e.g., in FIG. 1B, in blocks 202 - 212 within process 200 of FIGS. 2A - B, and in blocks 402 - 404 within process 400 of FIG. 4). The engine 608 may generate models for each quality metric / feature and / or for each food type. Although the present disclosure describes models for detecting the presence of infection and for determining the edibility of food, one or more other models may also be generated by using the same or similar techniques. Other models may include, but are not limited to, determining browning, spoilage, mold, ripening, hardness, taste, acidity level, Brix, size, shape, etc. of various types of food such as apples, avocados, berries, cucumbers, oranges, lemons, limes, and other citrus fruits.

[0151]

[0160] The models generated and trained by the model training engine 608 can be stored in the model data store 618 as models 622A - N. The models 622A - N can be accessed and / or retrieved by one or more analyzers of the quality evaluation engine 606 during runtime. Only some of the models 622A - N can be selected during runtime based on the type of food and / or user preferences in the image data for the quality evaluation of a specific food in the image data. The models 622A - N can also be updated or otherwise modified based on the runtime application of the models to the image data.

[0152]

[0161] The object detection engine 602 can be configured to detect one or more foods in the image data, such as block 304 in process 300 of FIGS. 3A - B, as described herein. The engine 602 can receive the image data and perform object detection techniques to process the image data. The engine 602 can also apply one or more machine learning trained models trained to identify any type of food or a specific type of food in the image data. The engine 602 can generate a bounding box around each of the foods in the image data. For each of the foods 624A - N, the engine 602 can store the bounding box image of the food in the food quality data store 620.

[0153]

[0162] In some implementations, the object detection engine 602 can calibrate the color within the bounding box portion of the image data to maintain color consistency across the entire image data. Color calibration can also be performed by a separate engine, such as a color calibration engine (not depicted). Color calibration can be applied to the entire image data as a pre - processing step before object detection and / or object extraction. When the image data is calibrated based on color, RGB image analysis as described throughout this disclosure can be performed.

[0154]

[0163] As described herein, the indexing engine 604 can be configured to apply a grid structure to image data and index each bounding box within the structure, for example, at block 306 within process 300 of FIGS. 3A - B. Each bounding box can enclose food represented within the image data. For each of the foods 624A - N, the engine 604 can retrieve a bounding box image from the food quality data store 620 and assign an index value to the bounding box. Thus, each food can be identified by an assigned grid index that can be useful for future retrieval, analysis, and processing operations of information. For each of the foods 624A - N, the engine 604 can store the grid index within the food quality data store 620.

[0155]

[0164] The quality assessment engine 606 may be configured to identify one or more characteristics indicative of the quality of the food represented within the image data. As described herein, the engine 606 may determine the presence of infection within the image data. One or more other quality assessment engines may also be part of the computer system 150. Each quality assessment engine, such as the quality assessment engines 132A - N of FIG. 1C, may be configured to determine different quality characteristics of the food. For example, the engine 606 may determine the presence of infection in any type of food, another engine may determine the presence of infection in an orange, another engine may determine the ripeness of an avocado, another engine may determine the firmness of an avocado, another engine may be a general color analyzer for any type of food, another engine may be a size analyzer, another engine may be an apple color analyzer, another engine may be an orange taste and / or juiciness analyzer, and so on. As another specific example, one or more quality assessment engines may include a color analyzer, an apple yellowing analyzer, a lime yellowing analyzer, an avocado internal analyzer, a lime spoilage and drying analyzer, a shelf life analyzer, a crop grading analyzer, a size analyzer, a shape analyzer, a ripening coating analyzer, a dry matter analyzer, a firmness analyzer, a Brix analyzer, a nutrition analyzer, a sticker analyzer, an acidity analyzer, a general internal defect analyzer, and / or a general color analyzer. One or more additional analyzers may include a wrinkle analyzer (e.g., for mangoes) and / or a skin oxidation (black spot) analyzer (e.g., for mangoes). The output from any one or more of the analyzers may be stored for each food 650A - N as a quality metric (or quality score) for each respective food 624A - N within the food quality data store 620. In some implementations, the output from any one or more of the quality assessment engines may be received as an input into one or more of the other quality assessment engines executed by the computer system 150 to determine the overall quality of a particular food or a batch of foods including the particular food.

[0156]

[0165] In FIG. 6, the quality evaluation engine 606 can retrieve the bounding box images of each of the foods 624A - N from the food quality data store 620. In some implementations, the engine 606 can receive the food bounding box images from the object detection engine 602. In some implementations, the engine 606 can directly retrieve the image data of a batch of foods from an imaging device or imaging sensor such as the imaging device 160 described in FIG. 1A and / or the imaging sensor 106 described in FIG. 1C.

[0157]

[0166] The food bounding boxes can be processed by the infection analyzer 612. The infection analyzer 612 can be configured to determine whether an infection is present within the food. The analyzer 612 can also determine whether an infection is present within a batch of foods based on an analysis of the bounding boxes of each food within the batch of foods. The analyzer 612 can retrieve one or more models 622A - N from the model data store 618 for execution during runtime. For example, the analyzer 612 can retrieve a model for detecting the presence of an infection as described throughout this disclosure. The analyzer 612 can also select a model from the model data store 618 based on the food type identification made by the object detection engine 602 and / or the metadata associated with the retrieved food bounding boxes of the foods (e.g., food information of each of the foods 624A - N within the food quality data store 620, which can include origin, geographical conditions of growth, transportation conditions, typical / historical ripening conditions, typical / historical growth conditions, food type, etc.). The analyzer 612 can apply a model to determine whether an infection is present within the food (as described in FIG. 1A, in blocks 216 - 218 within the process 200 of FIGS. 2A - B, and in blocks 310 - 314 within the process 300 of FIG. 3). The determination made by the analyzer 612 can also be stored within the food quality data store 620 for each of the foods 624A - N. This determination can be a quality metric as described herein.

[0158]

[0167] Quality assessment engine 606 may have an infection coverage determiner 614 configured to determine what percentage of the food within the bounding box is covered by an infection detected by the infection analyzer 612. The determiner 614 may generate a percentage value indicating the infection coverage of a particular food as described with reference to block 222 in process 200 of FIGS. 2A - B and block 312 in process 300 of FIGS. 3A - B. The determiner 614 may also generate one or more other values such as a string value, a boolean value, and / or a numerical value indicating what percentage of the food is covered by the infection / what percentage of the food contains the infection. The percentage of infection coverage or other value of infection coverage may be stored within the food quality data store 620 for each of the foods 624A - N.

[0159]

[0168] Quality assessment engine 606 may also have an edibility time length determiner 616 configured to determine the current edibility (e.g., consumption grading) and / or predict the remaining length of time that the food within the bounding box can be consumed by the end consumer / is edible. The edibility time length determiner 616 may also, in some implementations, predict the shelf life and / or ripeness of the food. Further, the determiner 616 may predict the total length of time of edibility of a batch of foods including a particular food within the bounding box. For additional discussion regarding the determination of the length of time of edibility, refer to process 400 in FIG. 4. Next, the length of time of edibility may be stored within the food quality data store 620 for each of the foods 624A - N.

[0160]

[0169] Finally, quality assessment engine 606 may generate an output regarding the quality of the food analyzed by components 612, 614, and 616. The engine 606 may also generate an output regarding the overall quality of a batch of foods including the particular food analyzed. This output may be sent to the user device 170 depicted and described in FIG. 1A.

[0161]

[0170] The supply chain modifier 607 can be configured to determine one or more supply chain modifications of a particular food or batch of foods based on quality metrics determined by the quality assessment engine 606. The supply chain modifier 607 can receive quality metrics (e.g., presence of infection, infection coverage, and / or length of time to edibility) from the quality assessment engine 606 (or can retrieve them from the food data records 624A-N in the food quality data store 620), and can also receive the supply chain schedule for the particular foods 624A-N from the food quality data store 620. Next, the supply chain modifier 607 can determine one or more modifications to the supply chain schedule based on the quality metrics, where these modifications can ensure a reduction in waste of the particular food or can otherwise ensure elimination of waste. For additional discussion regarding determination of supply chain modifications, see block 320 in process 300 of FIGS. 1A, 3A-B, and block 414 in process 400 of FIG. 4. Next, any modifications or updates to the supply chain schedule for each of the foods 624A-N can be stored in the food quality data store 620.

[0162]

[0171] FIG. 7 shows an example of a computing device 700 and an example of a mobile computing device that can be used to implement the techniques described herein. The computing device 700 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The mobile computing device is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are intended to be exemplary only, and thus are not meant to limit the implementations of the invention described and / or claimed in this document.

[0163]

[0172] Computing device 700 includes a processor 702, a memory 704, a storage device 706, a high-speed interface 708 that connects to the memory 704 and a plurality of high-speed expansion ports 710, and a low-speed interface 712 that connects to a low-speed expansion port 714 and the storage device 706. Each of the processor 702, the memory 704, the storage device 706, the high-speed interface 708, the high-speed expansion ports 710, and the low-speed interface 712 are interconnected by various buses and may be mounted on a common motherboard or in other manners as required. The processor 702 may process instructions for execution within the computing device 700, including instructions stored in the memory 704 or on the storage device 706 for displaying graphic information for a GUI on an external input / output device (such as a display 716 coupled to the high-speed interface 708). In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memory, as required. Also, multiple computing devices may be connected, such as each device providing a portion of the necessary operations (e.g., as a server bank, a single blade server, or a multiprocessor system).

[0164]

[0173] The memory 704 stores information within the computing device 700. In some implementations, the memory 704 is a volatile memory unit or units. In some implementations, the memory 704 is a non-volatile memory unit or units. The memory 704 may also be another form of computer-readable medium, such as a magnetic or optical disk.

[0165]

[0174] Storage device 706 can provide mass storage for computing device 700. In some implementations, storage device 706 can be, or can include, a computer-readable medium (such as a floppy disk device, hard disk device, optical disk device, or tape device, flash memory or other similar solid state memory device), or an array of devices (including a storage area network or other configured devices). A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods such as those described above. The computer program product can also be a computer-readable medium or machine-readable medium (such as memory 704, storage device 706, or memory on processor 702).

[0166]

[0175] High-speed interface 708 manages the bandwidth-intensive operations of computing device 700, while low-speed interface 712 manages the low-bandwidth intensive operations. Such an allocation of functions is merely exemplary. In some implementations, high-speed interface 708 is coupled to memory 704, to display 716 (e.g., via a graphics processor or accelerator), and to high-speed expansion port 710 that can receive various expansion cards (not shown). In this implementation, low-speed interface 712 is coupled to storage device 706 and low-speed expansion port 714. Low-speed expansion port 714, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices (such as a keyboard, pointing device, scanner, or networked devices such as switches or routers, e.g., via a network adapter).

[0167]

[0176] Computing device 700 can be implemented in many different forms, as shown in the figure. For example, computing device 700 can be implemented as a standard server 720, or can be implemented multiple times within a group of such servers. In addition, computing device 700 can be implemented within a personal computer such as a laptop computer 722. Computing device 700 can also be implemented as part of a rack server system 724. Alternatively, components from computing device 700 can be combined with other components within a mobile device (not shown) such as mobile computing device 750. Each of such devices can include one or more of computing device 700 and mobile computing device 750, and the entire system can be composed of multiple computing devices that communicate with each other.

[0168]

[0177] Mobile computing device 750 includes, among other components, a processor 752, a memory 764, input / output devices such as a display 754, a communication interface 766, and a transceiver 768. Mobile computing device 750 can also include a storage device such as a microdrive or other device to provide additional storage. Each of the processor 752, the memory 764, the display 754, the communication interface 766, and the transceiver 768 are interconnected by using various buses, and some of these components can be mounted on a common motherboard or in other ways as needed.

[0169]

[0178] Processor 752 can execute instructions within mobile computing device 750, including instructions stored in memory 764. Processor 752 can be implemented as a chipset of chips including separate and multiple analog and digital processors. Processor 752 can provide, for example, coordination of other components of mobile computing device 750 such as control of the user interface, applications executed by mobile computing device 750, and wireless communication by mobile computing device 750.

[0170]

[0179] Processor 752 can communicate with the user via control interface 758 and display interface 756 coupled to display 754. Display 754 can be, for example, a TFT (thin film transistor liquid crystal display) display or an OLED (organic light emitting diode) display, or other suitable display technology. Display interface 756 can include appropriate circuitry for driving display 754 to present graphic information and other information to the user. Control interface 758 can receive commands from the user and convert them into something for submitting to processor 752. Additionally, external interface 762 can provide communication with processor 752 to enable near area communication between mobile computing device 750 and other devices. External interface 762 can provide, for example, wireless communication in some implementations, or other implementations can provide wireless communication, and multiple interfaces can also be used.

[0171]

[0180] Memory 764 stores information within mobile computing device 750. Memory 764 can be implemented as a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. An expansion memory 774 can also be provided and can be connected to mobile computing device 750 via an expansion interface 772 (which can include, for example, a SIMM (Single In Line Memory Module) card interface). Expansion memory 774 can provide additional storage space for mobile computing device 750 or can also store applications or other information for mobile computing device 750. Specifically, expansion memory 774 can include instructions to execute or supplement the processes described above and can also include secure information. Thus, for example, expansion memory 774 can be provided as a security module for mobile computing device 750 and can be programmed with instructions to permit secure use of mobile computing device 750. In addition, a secure application (such as placing identification information on the SIMM card in a non-hackable manner) can be provided via the SIMM card along with additional information.

[0172]

[0181] As discussed below, the memory can include, for example, flash memory and / or NVRAM memory (non-volatile random access memory). In some implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods such as those described above. The computer program product can be a computer or machine-readable medium (such as memory 764, expansion memory 774, or memory on processor 752). In some implementations, the computer program product can be received, for example, in a propagated signal on transceiver 768 or external interface 762.

[0173]

[0182] The mobile computing device 750 can communicate wirelessly via a communication interface 766 that may include a digital signal processing circuitry configuration as needed. The communication interface 766 can provide communication under various modes or protocols (such as, among others, GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messages (Multimedia Messaging Service), CDMA (Code Division Multiple Access), TDMA (Time Division Multiple Access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), etc.). Such communication can occur, for example, via a transceiver 768 that uses radio frequencies. In addition, short - range communication can occur, such as by using Bluetooth, WiFi, or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 770 can provide additional navigation and location - related wireless data to the mobile computing device 750 as needed by applications running on the mobile computing device 750.

[0174]

[0183] The mobile computing device 750 can also communicate audibly by using a voice codec 760 that can receive spoken information from a user and convert it into usable digital information. The voice codec 760 can similarly generate audible sounds for the user, such as via a speaker (e.g., within the handset of the mobile computing device 750). Such sounds can include sounds from a voice telephone call, can include recorded sounds (such as voice mail, music files, etc.), and can also include sounds generated by applications operating on the mobile computing device 750.

[0175]

[0184] Mobile computing device 750 can be implemented in many different forms as shown in the figure. For example, mobile computing device 750 can be implemented as a cell phone 780. Mobile computing device 750 can also be implemented as part of a smartphone 782, a personal digital assistant, or other similar mobile devices.

[0176]

[0185] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuit configurations, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations within one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor coupled to receive and transmit data and instructions from and to a storage system, at least one input device, and at least one output device, which can be of a specific purpose or general purpose.

[0177]

[0186] These computer programs (also known as programs, software, software applications, or code) include machine language instructions for a programmable processor and can be realized in high-level procedural and / or object-oriented programming languages and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, or programmable logic device (PLD), etc.) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine language instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0178]

[0187] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, a keyboard, and a pointing device (e.g., a mouse or trackball) by which the user may provide input to the computer. Other types of devices may also be used to similarly provide interaction with the user: for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, tactile feedback); and the input from the user may be received in any form including acoustic, speech, or tactile input.

[0179]

[0188] The systems and techniques described herein may be implemented within a computer system that includes backend components (e.g., as a data server), or within a computer system that includes middleware components (e.g., an application server), or within a computer system that includes frontend components (e.g., a client computer having a graphical user interface or a web browser by which a user may interact with an implementation of the systems and techniques described herein), or within a computer system that includes any combination of such backend, middleware, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0180]

[0189] A computing system may include clients and servers. Clients and servers are typically remote from each other and typically interact via a communication network. The relationship between a client and a server arises by virtue of computer programs running on respective computers that have a client / server relationship to each other.

[0181]

[0190] This specification includes many detailed specific implementations, but these should not be construed as limitations on the scope of the disclosed technology or what may be claimed, but rather as descriptions of features that may be specific to the disclosed technology. Some features described herein with respect to separate embodiments may also be implemented, in part or in whole, in combination in a single embodiment. Conversely, various features described with respect to a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, some features may be described herein as acting in some combinations and / or may be so initially claimed, but one or more features from the claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a sub-combination or a variant of a sub-combination. Similarly, some operations may be described in a particular order, but this should not be understood as requiring that such operations be performed in that particular order or sequentially, or that all operations be performed to achieve the desired result. Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims.

Claims

[Claim 1] The invention described herein.

Images