Imaging system for object recognition and evaluation

Abstract

A method and system for using one or more sensors configured to capture two-dimensional and / or three-dimensional image data of one or more objects. In particular, the method and system combines one or more digital sensors with visible and near-infrared illumination to capture spectral image data of one or more objects in the visible and non-visible ranges. The captured spectral image data can be used to isolate and identify the one or more objects. Additionally, three-dimensional image data can be used to determine a volume of each of the one or more objects. The identification and volume data of the one or more objects can be used, individually or in combination, to obtain characteristics about the objects. The method and system provides a user with the ability to capture images of one or more objects and obtain relevant characteristics or information about each of the one or more objects.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and interest in the subject matter shared by both applications in co-pending U.S. Provisional Application No. 62 / 082,795, filed November 21, 2014. The entire disclosure of the provisional application is incorporated herein by reference. Technical Field

[0003] This invention generally relates to a system combining one or more sensors configured to capture two-dimensional and / or three-dimensional image data of one or more objects. In particular, the invention relates to a system configured to capture spectral image data and dimensional data of one or more objects, such as food items, and export this data to a web-based application. The web-based application portion of the system utilizes multiple algorithms and tabular data databases to facilitate the characterization of one or more objects, and implements comprehensive procedures to enable users to automatically track and monitor desired variables, properties, and characteristics of the one or more objects. Background Technology

[0004] Generally, people have always used a variety of different methods and devices to track diet, food consumption, and health. These methods include systems for counting calories, tracking points, excluding or restricting certain types of food, etc. These systems may also include health tracking to estimate how many calories a user may have burned with respect to an estimated calorie intake. Overall, the widespread use of smartphones, tablets, and other mobile devices has revolutionized how many people monitor and guide their food consumption habits.

[0005] However, these devices, systems, applications, and methods have several drawbacks. In particular, current dietary monitoring technologies suffer from several limitations, partly due to shortcomings in object imaging and characterization techniques. The most popular food-related tracking and recording apps are designed for use with smartphones, allowing users to download the app, create a user account, and input their weight loss goals, exercise levels, macronutrient intake, blood pressure, sleep patterns, etc. Conventional apps require users to manually record their food intake and exercise through self-reporting, based on selections from a web database of foods and associated values. When using manually reported information, dietary monitoring apps typically output nutritional information based on user input. Many users find manually inputting food information difficult, cumbersome, and confusing, leading to inconsistent or incorrect use of the app over time, thus reducing the effectiveness of this existing technology.

[0006] In addition, several handheld consumer spectrometers are more commonly used to analyze the chemical composition of food (primarily to identify allergens) using spectroscopy. However, these handheld spectrometers do not integrate any device for automatically determining the volume of individual portions or automatically separating and dividing the various foods on a plate to automatically analyze the entire meal. Spectrometers require users to manually input their dietary intake and exercise to monitor their calorie and nutrient intake. Furthermore, when users need to capture the chemical composition of food items in their diet, a handheld spectrometer represents another device that users carry with them for use, in addition to other commonly used devices such as smartphones. Finally, handheld consumer spectrometers are expensive. There is a need for a system that provides users with an integrated device to capture image data of one or more objects, such as food items, in a more cost-effective manner—utilizing existing user devices such as smartphones in a more robust way—and automatically evaluate the characteristics of those objects and use the identified characteristics to automatically perform desired processing, such as obtaining, tracking, and monitoring calorie and nutrient intake. Summary of the Invention

[0007] An imaging system is needed that allows a user to capture images of one or more objects, immediately correlate the captured images with data identifying the objects in the images, and calculate, determine, or obtain additional useful information about the objects. For the purposes of this application, the implementation of the imaging system will be described within the context of food item analysis, including providing the system user with the ability to capture images of food items and obtain features or information related to calories and nutrition (e.g., actual food weight) to enable diet management. However, as those skilled in the art will understand, the technology of this invention can be used in fields outside the field of food management.

[0008] In a specific embodiment of capturing food item characteristics, the system of the present invention enables a user to capture an image of one or more food items that the user will consume or purchase and immediately obtain a display of nutritional information and ingredient labels (e.g., similar to standard Food and Drug Administration (FDA) labels) on their device, showing the food's actual weight, calories, cholesterol, sodium, carbohydrates, sugar, and other nutritional values, and then record this information, for example, in the user's account.

[0009] The system of the present invention, in addition to having other desired features, also addresses other solutions to these needs. Specifically, the present invention is a comprehensive and automated system that integrates the functionality of a 3D scanning device and a two-dimensional and / or three-dimensional digital imaging sensor to capture the data required for the identification and characterization of at least one object, such as a food item, within the field of view. The system of the present invention provides a large set of cloud and / or network-based resources to utilize the data captured by the system to collect, compute, synthesize information about the one or more objects, and display such information to the user, such as the nutritional values ​​of the scanned food item in the exemplary implementation. The system of the present invention enables automated monitoring of food and nutrient intake. The system of the present invention can be used to collect information from the spectral characteristics of objects such as food items, two-dimensional or three-dimensional digital camera images, and 3D scans, and to combine this information with a web-based application communicating with multiple algorithmic and tabular object data databases to collect, store, and display information to the user, and in the food item embodiment, to provide such information related to the user's dietary habits and food intake.

[0010] According to an illustrative embodiment of the present invention, a method for automatically detecting and processing a plate of food for nutritional value is provided. The method includes: detecting the edges of the plate of food based on the depth of the food relative to other objects in a field of view using at least one sensor; and capturing a three-dimensional model of the food using the at least one sensor. The method further includes: capturing image data of the food using the at least one sensor, the image data including a visible light image and at least one near-infrared (NIR) image of the food; and converting the image data into a synthetic image using a processor, the synthetic image mimicking a single image captured by a single sensor. The method further includes: identifying food items corresponding to the synthetic image using the processor; converting the three-dimensional model of the identified food item into the volume of the identified food item using the processor; and calculating dietary information of the food item based on the volume of the identified food item using the processor.

[0011] According to an aspect of the invention, the method may include: determining an initial volume of the identified food item; determining a final volume of the identified food item; and calculating a volume change of the identified food item based on the difference between the initial volume and the final volume of the identified food item.

[0012] According to an aspect of the present invention, the method may include: obtaining dietary information of the identified food items from a database, and calculating the dietary content of the volume change of the identified food items.

[0013] According to an aspect of the invention, the method may include: illuminating the food on the plate with an LED array; obtaining reflectance image data at wavelengths of 745 nm, 810 nm, 940 nm, 970 nm and / or 1050 nm; and associating the reflectance image data with characteristic food elements.

[0014] According to an aspect of the invention, the volume and weight of the identified food item can be calculated with respect to the aforementioned visible light image and at least one near-infrared (NIR) image.

[0015] According to an illustrative embodiment of the present invention, a method for automatically detecting and processing one or more objects in a target region is provided. The method includes: detecting the edge of a target region based on the depth of the target region relative to other objects in a field of view using at least one sensor; and capturing a three-dimensional model of one or more objects in the target region using the at least one sensor. The method further includes: capturing image data of the one or more objects using the at least one sensor, the image data including RGB elements or vectors separated from a visible light image of reflected light from the one or more objects and a plurality of near-infrared vectors of specific wavelengths extracted from the reflected light image data of the one or more objects; and converting the image data into a synthetic image using an image processing module, the synthetic image simulating a single image captured by a single sensor. The method further includes: identifying at least one object among the one or more objects corresponding to color pixels in the synthetic image using the image processing module; and determining the spatial volume of each pixel in the three-dimensional model based on pixel depth using the image processing module. The method further includes: converting the spatial volume of each pixel of each identified at least one object into a volume value of the at least one object using the image processing module; and summing the volume values ​​of each of the color pixels of the identified at least one object in the synthetic image using the image processing module to calculate the total volume of the at least one object.

[0016] According to an illustrative embodiment of the present invention, a system is provided. The system includes a digital camera. The digital camera includes: at least one image sensor configured to convert captured visible light and near-infrared (NIR) light within a spectral range into captured photovoltages; at least one image processing module to convert the captured photovoltages into three-dimensional (3D) image data; and a recording device for recording the 3D image data. The digital camera captures and records at least two distinct and non-overlapping subsets of spectral ranges within the NIR spectral range, without recording spectral range gaps between the non-overlapping subsets. The system also includes an image processing engine. The image processing engine is configured to: analyze the captured and recorded photovoltages to identify one or more objects; determine volume data of the one or more objects over a given time period based on the recorded 3D image data; and obtain feature information data of the identified one or more objects from one or more databases. The system also includes a display device for outputting the volume data and feature information data of the one or more objects.

[0017] According to aspects of the invention, visible light can be captured in the spectral range of 400 nm to 700 nm, and NIR light can be captured in the spectral range of 700 nm to 1050 nm.

[0018] According to an aspect of the invention, the at least one image sensor may further include: a three-dimensional scanning device configured to capture three-dimensional point cloud data of the one or more objects; a digital NIR camera having a visible light blocking filter configured to capture photovoltage of the one or more objects; and an RGB digital camera configured to capture two-dimensional or three-dimensional image data of the one or more objects. The digital NIR camera captures a series of two or more images of the at least two different and non-overlapping subsets of spectral ranges by cyclically passing through the NIR LED array one at a time.

[0019] According to an aspect of the invention, the captured photovoltage is used to identify one or more sub-objects of the one or more objects. The identified one or more sub-objects of the one or more objects can be separated, and characteristic information data of each of the one or more sub-objects can be obtained from one or more databases.

[0020] According to an aspect of the invention, the artificial intelligence engine can identify at least one predefined primitive and trigger an RGB digital camera and a digital NIR camera to align with the at least one predefined primitive.

[0021] According to an aspect of the invention, the at least one image processing module can be configured to use three-dimensional point cloud data of one or more objects to determine the distance range from the at least one image sensor to the one or more objects, determine the surface area of ​​the one or more objects, and calculate the size and shape of the one or more objects using time of flight and surface area without the need for a reference marker. The time of flight and surface area of ​​the one or more objects can be determined by calculating the number of pixels in the three-dimensional point cloud data within the field of view.

[0022] According to an illustrative embodiment of the present invention, a system for automatically identifying an object is provided. The system includes: a near-infrared (NIR) light-emitting diode (LED) array comprising at least two different wavelengths of LEDs; a three-dimensional scanning device configured to capture a three-dimensional image of the object; a digital RGB camera configured to capture two-dimensional and / or three-dimensional visible light image data of the object; and a digital NIR camera configured to capture NIR image data of the object. The NIR LED array emits controlled light within a predetermined spectral range, and the three-dimensional scanning device captures three-dimensional image data of the object, the digital RGB camera captures two-dimensional visible light image data of the object, and the digital NIR camera captures a series of NIR datasets, the capture of which is performed by triggering a unique LED wavelength for each NIR dataset of the object. The three-dimensional image data of the object, the two-dimensional and / or three-dimensional visible light image data of the object, and the NIR data of the object are converted by the system into characteristic definitions related to the composition and volume of the object.

[0023] According to an illustrative embodiment of the present invention, a system is provided. The system includes a digital camera. The digital camera includes at least one image sensor configured to convert captured visible light in one spectral range and near-infrared (NIR) light in another spectral range into captured image data. The system also includes at least one three-dimensional image scanning device configured to capture three-dimensional image data; wherein the digital camera captures at least two distinct and non-overlapping subsets of spectral ranges within the NIR spectral range, without capturing light in the spectral range gaps between the non-overlapping subsets of spectral ranges.

[0024] According to aspects of the invention, the at least one image sensor may further include: a three-dimensional scanning sensor configured to capture three-dimensional image data; a digital RGB camera sensor configured to capture two-dimensional visible light image data; and a digital NIR camera sensor having a visible light blocking filter, the digital NIR camera sensor being configured to capture two-dimensional and / or three-dimensional NIR image data. In some implementations, the at least one image processing module may be configured to convert image data from the three-dimensional scanning sensor, the digital RGB camera sensor, and the digital NIR camera sensor into a representative image dataset originating from a single sensor. In some implementations, the system may further include at least two light source devices configured to generate non-overlapping spectra. The at least one image sensor captures image data with respect to at least two different and non-overlapping subsets of spectral ranges within the NIR spectral range for each of the at least two light source devices during the activation of each individual light source of the at least two light source devices.

[0025] According to an aspect of the invention, the at least one image processing module can be configured to capture N image datasets. The N image datasets include image data from a visible light image capture and image data from N-1 NIR image captures. The at least one image processing module can also be configured to capture ambient image data and subtract the ambient image data from the N image datasets when all at least two light source devices are set to off.

[0026] According to an aspect of the invention, the digital camera further includes a lens, a sensor, and a three-dimensional microprocessor, and the at least one sensor captures three-dimensional image data using at least one of stereo vision, time-of-flight, and structured light.

[0027] According to aspects of the invention, the at least one image sensor may further include at least a microelectromechanical spectroscopy chip set, an infrared (IR) source, a plurality of focusing lenses, a slit, an IR bandpass filter, a diffraction grating, a digital micromirror device, a detector, and a microprocessor. The system may also include wired and wireless connectivity devices configured to transmit data over a network.

[0028] According to aspects of the invention, a digital camera can be configured to convert captured ultraviolet (UV) light in the spectral range of 100 nm to 400 nm into a captured photovoltage. In some implementations, visible light can be captured in the spectral range of 400 nm to 700 nm, and NIR light can be captured in the spectral range of 700 nm to 1050 nm.

[0029] According to an illustrative embodiment of the present invention, an object evaluation system is provided. The system may include an imaging sensor application programming interface (API) configured to capture and store visual spectral range image data and near-infrared spectral image data of one or more objects. The system may also include a three-dimensional scanning device API configured to capture and store three-dimensional image data of the one or more objects. The system may further include a visual comparison API configured to identify the one or more objects based on the captured visual spectral range image data and near-infrared spectral image data. The system may also include an automatic segmentation API configured to separate each of the one or more objects based on the identification via the visual comparison API, and to calculate the volume of each of the separated one or more objects based on the three-dimensional image data.

[0030] According to an aspect of the invention, the one or more objects can be one or more food items in a diet. The system may also include a volume and weight API configured to compare the identified one or more objects against a remote database to determine the nutritional value of the one or more objects. The system may also include a nutritional value output API configured to output nutritional information for each of the one or more objects based on its nutritional value.

[0031] According to an aspect of the present invention, the visual comparison API can compare visual spectral range image data and near-infrared spectral image data with predetermined image data values ​​stored in a database.

[0032] According to an aspect of the invention, the system may further include a barcode and optical character recognition API configured to analyze a Universal Product Code (UPC) or a character list to obtain additional information for the system.

[0033] According to an illustrative embodiment of the present invention, a method is provided. The method includes capturing a three-dimensional image using a three-dimensional image scanning device. The method further includes determining a distance length between the three-dimensional image scanning device and one or more objects based on data from the three-dimensional image. The method further includes determining the volume of each of the one or more objects using an image processing module, based on the distance length and the data from the three-dimensional image. The method also includes capturing a visible light image using an RGB camera module; and capturing a sequence of NIR images using a near-infrared (NIR) camera module, the capture of the NIR image sequence being performed while simultaneously triggering a unique LED wavelength for each captured NIR image. The method further includes analyzing the captured visible light image and the NIR image sequence using an artificial intelligence module to identify each unique object among the one or more objects; and determining additional features about the one or more objects based on the identified one or more objects and the volume of each of the one or more objects.

[0034] According to one illustrative embodiment of the invention, an imaging system is provided for object identification and evaluation and equivalent object identification and evaluation as described herein, in any operable combination.

[0035] According to one illustrative embodiment of the present invention, a method is provided for identifying and evaluating the objects and equivalents described herein in any combination of operable steps. Attached Figure Description

[0036] These and other features of the invention will be more fully understood by referring to the following detailed description in conjunction with the accompanying drawings, in which:

[0037] Figure 1 A system according to the invention for capturing and automatically determining the characteristics of one or more objects is described;

[0038] Figure 2 A system for capturing and automatically determining the characteristics of one or more objects—implemented as food item objects—according to aspects of the invention is described;

[0039] Figure 3A , Figure 3B , Figure 3C and Figure 3D An exemplary housing for a system according to an aspect of the invention is shown;

[0040] Figure 4 It is a graphical representation of the raw spectral data captured, interpreted, and used by the system according to aspects of the present invention;

[0041] Figure 5It is a graphical representation of spectral feature data derived from captured image data and used by the system according to an aspect of the invention;

[0042] Figure 6 According to an aspect of the invention, it includes a cloud map of many pixel data points obtained using the system;

[0043] Figure 7 This is an illustrative flowchart depicting an exemplary process of using a system according to aspects of the present invention for food item evaluation;

[0044] Figure 8 This is an illustrative flowchart describing an exemplary process of using the system according to aspects of the present invention for food item evaluation;

[0045] Figure 9 This is a description of the diet to be captured and processed according to an aspect of the present invention;

[0046] Figure 10A and Figure 10B According to aspects of the present invention, scanning Figure 9 The description of nutritional information obtained from the diet; and

[0047] Figure 11 This is a diagrammatic illustration of a high-level architecture for implementing systems and processes according to aspects of the present invention. Detailed Implementation

[0048] The illustrative embodiments of the present invention relate to a system for imaging one or more objects and simultaneously determining the characteristics (e.g., identification, analysis, etc.) of the one or more objects, and obtaining additional information about the one or more objects, at least in part, based on the identification and volume determination of the one or more objects, without using a reference object as a scale. In particular, the present invention relates to a system using both visible light imaging and near-infrared three-dimensional imaging for identifying and analyzing one or more objects, such as food items, within a field of view, and determining the volume and other characteristics of those objects, such as the calories and nutritional value of the food items. If desired, the technology of the present invention enables the localization of multiple objects (e.g., a whole plate of food) within a single field of view and the implementation of the process of the present invention to image and determine the characteristics of those multiple objects. The present invention utilizes a single two-dimensional and three-dimensional sensor or a combination of a sensor and one or more light sources to capture image data (or captured photovoltage) of the one or more objects using most of the electromagnetic spectrum. The system of the present invention provides a unique combination of different sensing methods combined using sensor fusion techniques. Specifically, two-dimensional and three-dimensional sensors are capable of capturing image data within the electromagnetic wavelength range covering the visible spectral region of approximately 400 nm to 700 nm and the invisible near-infrared spectral region of approximately 700 nm to 1050 nm. Identification of one or more objects can be achieved by combining image analysis of image data in the visible spectral range with spectral analysis of image data in the near-infrared spectral region. Spectral analysis can be performed on the data captured by two-dimensional and three-dimensional sensors to obtain the spectral characteristics of one or more objects within the captured images, and these spectral characteristics can be used to identify one or more objects within the captured images.

[0049] Additionally, two-dimensional and three-dimensional sensors can capture three-dimensional data of an object and use the captured three-dimensional data to determine the estimated volume of the object or more. Combined spectral features and three-dimensional volume data can be captured and analyzed to automatically determine various characteristics of one or more target objects. In one exemplary embodiment, combined visual image data, spectral feature data, and three-dimensional volume data of food items can be used to automatically determine the actual weight of the food, separate and divide various foods on a plate, and determine the standard nutrient label for each food item in the captured dietary image. Furthermore, the three-dimensional data can be used to determine the difference in volume data over different time periods. The difference in volume data can be applied to various useful purposes, including, for example, determining the amount of food items consumed by a user and the associated nutrients of that amount of food items.

[0050] When used in conjunction with the dietary system of this invention, combined two-dimensional visual image data, spectral feature image data, and three-dimensional volumetric data can be used to automatically track a user's nutritional intake and provide additional information to help achieve nutritional goals. One implementation of automatic nutrition tracking requires the user to capture image data and three-dimensional volumetric data before consuming the diet, and then capture them again afterward. The system of this invention can use image data and three-dimensional volumetric data to automatically identify food items in the diet, separate various types of food items, determine the initial and final food volumes for each type, and calculate the nutritional value of the consumed portion of the diet (e.g., the nutritional value of the difference between the initial and final food volumes). This process provides the user with all the nutritional information about the consumed food, requiring only image data of the diet to be captured before and after consumption, without the need for baseline data or manual data input by the user. The two-dimensional and three-dimensional sensors used in the object assessment system / diet tracking system, and the unconventional steps involved, provide a unique improvement to object recognition technology. In particular, the unconventional steps of using data captured by an image sensor to identify individual items in a set of items, determining the volume data of each of the identified items, and determining additional characteristic information about the identified items based on the identification and volume are improvements to conventional object recognition techniques. As those skilled in the art will understand, embodiments that track the nutritional information of identified food items are intended for illustrative purposes only, and the present invention can be used for specific purposes to identify and calculate volume information and other characteristic information of various types of objects.

[0051] Figures 1 to 11 —wherein the same portions throughout these figures are indicated by the same reference numerals—illustrates one or more exemplary embodiments of a system and method according to the invention for capturing spectral characteristics and three-dimensional volumetric data of one or more objects and determining various features of such one or more objects. While the invention will be described with reference to one or more exemplary embodiments shown in the accompanying drawings, it should be understood that many alternative forms may be implemented. Those skilled in the art will also understand that different methods of altering parameters of the disclosed embodiments—such as the size, shape, or type of elements or materials—still remain within the spirit and scope of the invention. Furthermore, it should be noted that, unless otherwise stated, all references to specific wavelengths expressed herein, for example in nanometers (nm), in the detailed description, figures, and claims are intended to include the specified exemplary wavelength plus or minus 10%, as readily understood by those skilled in the art.

[0052] Figure 1 An illustrative system for carrying out aspects of the invention is described. In particular, Figure 1A system 100 including an object assessment system 102 is depicted. According to an exemplary embodiment, the object assessment system 102 may be a network-connected or cloud-connected computing infrastructure that provides tools for capturing image data of one or more objects, converting the image data, identifying the one or more objects in the converted image data, and providing additional features about the one or more objects. As those skilled in the art will understand, the image data may include any information that can be obtained and / or derived from the captured digital images (e.g., two-dimensional or three-dimensional images). For example, the object assessment system 102 may include a cloud-based application designed to identify one or more food items, determine the volume of the food items, and determine the nutritional value of the food items based on their identification and volume. Additionally, the object assessment system 102 may use image data of food items before and after a meal to estimate a user's nutritional intake.

[0053] The object evaluation system 102 may include a computing device 104, input and output devices 112, and a storage system 114, the computing device having a processor 106, memory 108, and input / output interfaces 110. As those skilled in the art will understand, the computing device 104 may include a single computing device, a group of computing devices in a network computing system, cloud computing infrastructure, or a combination thereof. Similarly, as those skilled in the art will understand, the storage system 114 may include any combination of computing devices configured to store and organize a large amount of data. For example, the storage system 114 may be a local storage device on the computing device 104, a remote database facility, or a cloud computing storage environment. The storage system 114 may also include a database management system utilizing a given database model, configured to interact with users to analyze database data.

[0054] continue Figure 1 The object evaluation system 102 may include a combination of core modules configured to perform various functions of the present invention. According to an exemplary embodiment of the present invention, the object evaluation system 102 may include at least an image processing module 116 and an artificial intelligence module 118. The image processing module 116 and the artificial intelligence module 118 may be any combination of software and hardware configured to perform aspects of the present invention. For example, the image processing module 116 may be configured to capture image data and perform operations on the image data, and the artificial intelligence module 118 may be configured to analyze the generated image data received from the image processing module 116.

[0055] According to an exemplary embodiment of the present invention, the input and output device 112 may include an imaging sensor 120 and a light source device 122, or otherwise communicate with the imaging sensor and the light source device. The imaging sensor 120 and the light source device 122 may include any combination of sensors or devices capable of capturing images and image data and providing illumination for capturing those images. For example, the imaging sensor 120 may include a single sensor capable of capturing visible light image data, infrared image data, short-wave infrared image data, and near-infrared two-dimensional and / or three-dimensional image data of an object. According to an exemplary embodiment of the present invention, the imaging sensor 120 may include any combination of a digital red-green-blue (RGB) camera sensor, a near-infrared sensor (NIR), a three-dimensional scanning device, etc., configured to capture and analyze visual image data, spectral feature image data, and three-dimensional image data of an object. For example, the imaging sensor 120 may capture image data defined by the absorption or emission characteristics of the wavelength of light incident on the surface of an object. Similarly, the light source device 122 may include any combination of light sources capable of generating illumination for the imaging sensor 120 to capture visible light image data, invisible light image data, and near-infrared two-dimensional and / or three-dimensional image data of an object. For example, the light source device 122 may include an array of light-emitting diodes (LEDs) capable of generating appropriate illumination for capturing visual image data, spectral feature image data, and three-dimensional image data by the imaging sensor 120. According to an exemplary embodiment of the invention, the imaging sensor 120 and the light source device 122 may be separate devices attached to or communicating with the rest of the object evaluation system 102. For example, the imaging sensor 120 and the light source device 122 may be included within the computing device 104, or within a separate input and output device 112 communicating via the I / O interface 110 of the computing device 104.

[0056] According to an exemplary embodiment of the present invention, the object evaluation system 102 may be configured to communicate with other computing devices 124 or components of system 100 (e.g., storage system 114, imaging sensor 120, etc.) via a wireless network such as a telecommunications network 126. The other computing devices 124 may be implemented as part of the object evaluation system 102, and their tasks may be configured to perform any combination of data acquisition, computation, transformation, analysis, and data output. According to an exemplary embodiment of the present invention, the object evaluation system 102 may be integrally embedded within and cooperate with other computing devices 124 to perform various aspects of the present invention. As those skilled in the art will understand, the other computing devices 124 may include any combination of computing devices—as described with respect to the computing devices of the object evaluation system 102. For example, the other computing devices 124 may include personal computers, laptop computers, tablet computers, smartphones, etc. According to an exemplary embodiment of the present invention, the other computing devices 124 may be configured to establish a connection and communicate via a telecommunications network 126 to perform various aspects of the present invention. As those skilled in the art will understand, the telecommunications network 126 may include any combination of wireless networks. For example, telecommunications network 126 may be a combination of mobile networks, WANs, LANs, Bluetooth®, or other types of wireless network technologies. Telecommunications network 126 may be used to exchange data between computing devices, exchange data with storage system 114, and / or collect data from other data sources.

[0057] Figure 2 An illustrative architecture for implementing exemplary aspects of the invention is depicted. In particular, Figure 2 Describing the implementation as about Figure 1 An exemplary system 200 of the object evaluation system 102 discussed herein. System 200, having object evaluation system 102, is configured to capture two-dimensional and / or three-dimensional image data of one or more objects, analyze the captured image data, identify the one or more objects, determine the dimensions of the one or more objects, and obtain additional features about the one or more objects. For example, system 200 can be used to analyze a diet to identify individual food items in the diet, determine the dimensions of the food items (e.g., surface area, volume, etc.), and calculate the nutritional value and chemical composition of the diet based on the identified food items and their corresponding dimensions. Figure 2The depicted object evaluation system 102 includes a computing device 104 housing components necessary to perform the functions of the present invention. According to an exemplary embodiment of the invention, the computing device 104 includes a microprocessor 106 to at least process the capture of image data. As those skilled in the art will understand, the microprocessor 106 can perform some or all of the steps of capture, calculation, determination, analysis, and conversion according to the invention. The computing device 104 also includes a plurality of imaging sensors 120 communicatively attached to the computing device 104. The plurality of imaging sensors 120 may include a near-infrared (NIR) camera module 120a, a visible light camera module 120b (e.g., an RGB camera module), and a three-dimensional scanning device 120c.

[0058] NIR camera module 120a can be any sensor capable of capturing spectral characteristic image data in the near-infrared region of the electromagnetic spectrum. For example, NIR camera module 120a can be a digital NIR camera sensor with a visible light blocking filter (e.g., cutoff at 700 nm+) capable of capturing spectral characteristic image data in the electromagnetic wavelength spectrum from approximately 700 nm to 1050 nm. An example of a digital NIR camera sensor is the Omnivision 5647 sensor equipped with a visible light blocking filter such as the Wratten 88A (or a similar filter with a 700 nm cutoff). Similarly, visible light camera module 120b can be any sensor capable of capturing visible light image data in the visible wavelength spectral region (400 nm to 700 nm). For example, visible light camera module 120b can be a digital RGB camera capable of capturing image data in the spectrum from approximately 400 nm to 750 nm. An example of a digital RGB camera sensor is the Omnivision 5647 sensor with a NIR blocking filter and a fixed-focus module. As those skilled in the art will understand, there are many combinations of wavelengths that can be included, and the wavelengths presented are merely examples. Additionally, the visible light camera module 120b can also capture the shape and texture data of an object. As those skilled in the art will understand, the NIR camera module 120a and the visible light camera module 120b are capable of capturing RGB images of different spectral wavelengths. Three-dimensional image scanning device

[0059] According to an exemplary embodiment of the present invention, the 3D scanning device 120c can be any sensor device capable of capturing 3D image data of an object or modeling the object. For example, the 3D scanning device 120c may include a laser, laser diode and sensing device, and a 3D microprocessor configured to capture and analyze 3D image data. According to an exemplary embodiment of the present invention, the 3D scanning device 120c can capture 3D image data without a reference / reference object. As those skilled in the art will understand, any combination of stereo vision, time-of-flight, structured light methods, or any methods known in the art can be used to capture 3D data. For example, the 3D microprocessor of the 3D scanning device 120c can analyze the distance or distance length from the 3D image scanning device 120c to the object for the captured image (e.g., by using time-of-flight) and use the captured image data and the analysis of the distance length to create point cloud data output (e.g., in .txt or .asc file format). The 3D image data may include data related to the shape and volume of the object. As those skilled in the art will understand, a single sensor device can be used to perform all the functions discussed regarding camera modules 120a, 120b, and 120c.

[0060] According to an exemplary embodiment of the present invention, computing device 104 includes a light source device 122 to provide necessary illumination for imaging sensor 120 to capture various spectral image data. As those skilled in the art will understand, light source device 122 may include any illumination device capable of generating the necessary illumination to capture data in the visible light spectral region and the invisible light / NIR spectral region. According to an exemplary embodiment of the present invention, light source device 122 may be an array of light-emitting diodes (LEDs) comprising a variety of different colors to simulate illumination of various spectra. For example, the LED array may include illumination for five different near-infrared spectral wavelengths in the range of about 700 nm to 1100 nm, and illumination for white light in the visible spectrum ranging from about 400 nm to 700 nm. As those skilled in the art will understand, there are many combinations of wavelengths that can be included, and the wavelengths presented are merely examples. In addition, the LED array may be adjusted by adding or subtracting 10% from the center wavelength in the nanometer range, which may be compensated for as needed by computing device 104.

[0061] According to an exemplary embodiment of the present invention, computing device 104 may include other components communicatively attached to microprocessor 106 and capable of providing data for further processing and analysis. For example, computing device 104 may include wired and / or wireless communication interfaces (e.g., WiFi, Bluetooth®, cellular, Universal Serial Bus (USB), etc.), geolocation interfaces (e.g., Global Positioning System (GPS)), power supply, microphone, display, speaker, motion sensing devices, etc. The wired and / or wireless communication interfaces can be used to transmit information from computing device 104 to network-based applications (e.g., applications on object evaluation system 102, storage system 114, other computing devices 124, etc.). For example, if object evaluation system 102 is implemented as a cloud-based application or installed on a local computing device (e.g., laptop computer, desktop computer, etc.), computing device 104 can communicate with the rest of object evaluation system 10 via wired or wireless interfaces. Similarly, for example, computing device 104 can communicate directly with intermediate computing devices (e.g., smartphones, tablets, desktop computers, laptops, etc.) via wired wireless communication interfaces, and the intermediate computing devices can transmit data to remote object evaluation system 102.

[0062] According to an exemplary embodiment of the present invention, a geolocation interface can be used to obtain location information of a computing device 104 to be provided to an object evaluation system 102. According to an exemplary embodiment of the present invention, geolocation information can be obtained via GPS and used by the object evaluation system 102 to collect additional information about a specific object. For example, geolocation information can be used to obtain a regional menu of a specific chain restaurant when determining the nutritional elements of an identified food item. As those skilled in the art will understand, the computing device 104 can be communicatively attached to or embedded in another device and utilize the wireless interface, microprocessor, and / or geolocation interface of the attached host device. For example, the computing device 104 can be a mobile computing device case inserted into a mobile computing device (e.g., a smartphone), and the computing device 104 can utilize the hardware and software of the mobile computing device to perform wireless communication, processing, geolocation, audio-visual presentation, etc. According to an exemplary embodiment of the present invention, system 200 may also include another computing device 124 and a database 126 (e.g., storage system 114), as per an example of... Figure 1 The subject of discussion.

[0063] Figures 3A to 3D Depicting the use of accommodating about Figure 1 and Figure 2 Various views of the housing implementation 300 of the computing device 104 under discussion. In particular, Figures 3A to 3DA housing implementation 300 is depicted surrounding other components of the computing device 104, such as imaging sensors 120a and 120b, a light source device 122, wired and / or wireless communication interfaces, a geolocation interface, a power supply, a microphone, a display, a speaker, motion sensing devices, etc. According to an exemplary embodiment of the invention, the housing implementation 300 may have approximately the same or similar thickness and width as commonly used mobile computing devices (e.g., smartphones). For example, the height of the housing implementation 300 may be approximately five and three-quarters inches. As those skilled in the art will understand, the housing implementation 300 may include a single device, a combination of modules, or a single peripheral device to be connected to another computing device.

[0064] According to an exemplary embodiment of the present invention, the housing implementation device 300 may include a mounting system. This mounting system enables the computing device 104 to be attached to the surface, underside, or surrounding another object, wherein the location of the computing device facilitates user access to the object evaluation system 102 and the computing device 104. The mounting system may include a mobile computing device box and / or a cabinet bracket. For example, a cabinet bracket enables the device to be attached to the underside of a kitchen cabinet, allowing for easy capture and analysis of information related to food prepared in the kitchen. According to an exemplary embodiment of the present invention, the computing device 104 may be directly integrated with commonly used mobile computing devices. For example, the housing implementation device 300 surrounding the computing device 104 may serve as a mobile computing device box for the mobile computing device and may be communicatively attached to another mobile computing device 124 (e.g., wirelessly or via the input / output port of the mobile computing device).

[0065] According to an exemplary embodiment of the present invention, computing device 104 may be integrally embedded within another existing computing device (e.g., another computing device 124). Figure 3D An exemplary representation of a computing device 104 embedded within another computing device is depicted. Specifically, Figure 3D A computing device 104 is depicted embedded within a smartphone device. Therefore, the housing implementation device 300 is another computing device 124, and the computing device 104 is integrated within the housing implementation device 300 (e.g., the other computing device 124) and is communicatively attached to the housing implementation device.

[0066] According to an exemplary embodiment of the present invention, computing device 104 can be used to enable a mobile computing device (e.g., another mobile computing device 124) to operate within object evaluation system 102. Specifically, housing implementation device 300 may include imaging sensors 120a and 120b, a light source device 122, a wired and / or wireless communication interface, a geolocation interface, a power supply, a microphone, a display, a speaker, and motion sensing devices for use by another mobile computing device 124. The wired communication interface may include a pass-through USB, multiple adapters, and a cover. The pass-through USB may also include a pair of female USB input terminals. The adapter may include any combination of male and female USB plugs and may be used to allow computing device 104 to communicate with other computing devices 124. For example, the USB plug may be used to connect I / O interface 110 to another mobile computing device 124, such as a smartphone or tablet, via a wired connection. When the wired connection is not used, the cover may be inserted into the USB input terminals of the device. According to an exemplary embodiment of the present invention, the housing implementation device 300 may include a plurality of external lenses or windows 302 for various image capture components (e.g., imaging sensor 120 and light source device 122).

[0067] continue Figures 3A to 3D The housing implementation device 300 may include an actuation mechanism 304. The actuation mechanism 304 may include one or more buttons to enable a user to trigger various components of the computing device 104 by activating the actuation mechanism 304 when the device approaches one or more target objects. Those skilled in the art will understand that the exact location and mechanism of the actuation mechanism 304 on the housing implementation device 300 may vary and are not limited to the location and mechanism shown in the figures. According to an exemplary embodiment of the invention, the actuation mechanism 304 may be associated with a motion detection sensor such that the functionality of the computing device 104 is activated when a user swings the computing device 104 over one or more target objects. For example, a user may “swing” the computing device 104 over a dish containing food items to be consumed, and the actuation mechanism 304 may be triggered by the motion detection sensor to initiate the imaging sensor 120 and subsequent processing steps, as discussed in more detail herein. According to an exemplary embodiment of the invention, the housing implementation device 300 may include a power source. The power source provides current to the various components of the computing device 104 of the invention. The power source may include a battery capable of being recharged via a wired connection, a wired power source via a USB input, or any power source known in the art.

[0068] During operation, such as regarding Figure 1 and Figure 2The object evaluation system 102 discussed in systems 100 and 200 described herein can be used to automatically capture two-dimensional and three-dimensional image data (e.g., visible light image data, spectral feature image data, and three-dimensional data) of one or more objects and convert the image data into a format for identifying the one or more objects and determining their volume. Subsequently, the object evaluation system 102 can use the identification and volume of the one or more objects to determine other desired characteristics of the one or more objects. In an exemplary implementation, the object evaluation system 102 can be used to capture two-dimensional and three-dimensional image data of a diet and use this image data to identify different food items in the diet, calculate the dimensions (e.g., volume, surface area, weight, etc.) of each food item, and determine the nutritional value of the entire diet by summing the nutritional values ​​of each food item based on its corresponding volume.

[0069] According to an exemplary embodiment of the invention, the entire process of the invention is initiated by targeting one or more objects using the imaging sensor 120 of the computing device 104 and capturing a combination of visible light images, visible near-NIR images, and three-dimensional images under a corresponding combination of illumination. The one or more objects can be any object for which the object evaluation system 102 is calibrated to identify, and for which the user is interested in obtaining additional information (e.g., size, features, etc.) about it. For example, the computing device 104 and the object evaluation system 102 can be calibrated and used to identify food items within the field of view, determine the volume of each food item, and subsequently provide nutritional information about the food items based on their volume. As those skilled in the art will understand, calibration can include changing a combination of hardware and software settings used to identify a particular type of object. For example, the hardware can include a light source device 122 required to generate appropriate illumination conditions for capturing images of a desired spectral range of a particular type of object using the imaging sensor 120. According to an exemplary embodiment of the invention, the object evaluation system 102 can be calibrated to capture specific properties of one or more objects. For example, object evaluation system 102 can be calibrated to capture characteristic food elements using a specific spectral wavelength of 745 nm as a first baseline, 810 nm as a second baseline, 940 nm for fats, 970 nm for carbohydrates / water, and 1020 / 1050 nm for proteins. As those skilled in the art will understand, the identified wavelengths are for illustrative purposes only, and object evaluation system 102 can be configured to capture any combination and range of spectral wavelengths known in the art, wherein the specific wavelength is determined based on one or more objects being evaluated.

[0070] The computing device 104 can use a combination of imaging sensors 120 to capture two-dimensional and / or three-dimensional digital images of one or more objects within a field of view. According to aspects of the invention, image data can be extracted from the digital images for use. As those skilled in the art will understand, the image data can include any combination of data that can be obtained and / or derived from the digital images captured by the imaging sensor 120. According to an exemplary embodiment of the invention, the imaging sensor 120 can acquire image data of the one or more objects covering a variety of different electromagnetic wavelengths. For example, the sensor 120 can be a combination of an NIR camera module 120a configured to capture image data of invisible near-infrared spectral wavelengths (e.g., about 700 nm to 1050 nm) and a visible light camera module 120b configured to capture image data of visible wavelengths (e.g., about 400 nm to 650 nm). Alternatively, the imaging sensor 120 can be configured to capture three-dimensional image data or a three-dimensional model of each of the one or more objects. For example, the imaging sensor 120 can include a three-dimensional scanning device 120c that can be used to capture three-dimensional image data of the one or more objects or to construct a three-dimensional model of the one or more objects. As those skilled in the art will understand, a single imaging sensor or multiple imaging sensors can be used to obtain image data discussed with respect to imaging sensors 120a, 120b, and 120c.

[0071] When using more than one imaging sensor 120 to capture the desired image data, the object evaluation system 102 can take additional processing steps for sensor fusion (e.g., using the microprocessor 106). According to an exemplary embodiment of the invention, digital images (e.g., image data) from multiple imaging sensors 120 can be combined or transformed to create a single representation of the overall image data using any method known in the art. For example, the object evaluation system 102 can utilize sensor fusion to convert multiple sensor image data inputs into a series of vectors defining the response surface of each of the captured images of one or more objects as varying with spectral wavelength. Thus, each element of the one or more objects can be represented by a spectral function defined by the IR reflectance response and the visible spectral response of visible light (e.g., color). As those skilled in the art will understand, if a single imaging sensor is used to capture all the imaging data, no sensor fusion processing step is required. Sensor fusion can be performed by any combination of the microprocessor 106, the image processing module 116, or other cloud computing systems.

[0072] According to an exemplary embodiment of the present invention, in operation, the light source device 122 can be configured to provide the illumination required for the two-dimensional and three-dimensional image data captured by the respective imaging sensors 120 in the desired spectral wavelength range. The light source device 122 can be any light source known in the art that can provide illumination for images captured in various spectral wavelength ranges. For example, the light source device 122 may include an array of light-emitting diodes (LEDs) capable of generating illumination for capturing images in both visible and invisible near-infrared spectral wavelengths. The light source device 122 can generate a white light source for capturing visible spectral image data (e.g., RGB), and can use a series of different non-overlapping illuminations for invisible near-infrared spectral image data. As those skilled in the art will understand, the light source device 122 can generate the necessary illumination by light generated from individual red, green, and blue LEDs, either in place of white light source LEDs or in combination with white light source LEDs.

[0073] According to an exemplary embodiment of the invention, an LED array can be used to generate illumination conditions by cyclically passing LEDs of specific wavelengths configured to produce each of the non-overlapping target wavelengths, and to capture image data at each non-overlapping wavelength. For example, the LED array may include five LEDs, and the imaging sensor 120 may capture image data with each LED on while the other four LEDs are off. As those skilled in the art will understand, the LED array or alternative light source may include any number of LEDs required to create the desired illumination environment for capturing image data within the desired non-overlapping spectral wavelength range.

[0074] According to an exemplary embodiment of the present invention, the LED array may include five NIR LEDs and three visible red, blue, and green (RGB) LEDs. The array of eight LEDs can be used in such a way that the individual LEDs are uniquely modulated using a Fourier transform method to simultaneously acquire eight wavelengths (e.g., spectral image data generated by the five NIR LEDs and visible light image data generated by the three RGB LEDs). Simultaneous acquisition of eight wavelengths reduces spectral acquisition to approximately one second while minimizing the influence of ambient light. In this exemplary embodiment, image acquisition can be synchronized with the modulated wavelengths (e.g., via image processing module 116) to separate the environment (unmodulated) from the spectral signal (modulated). As those skilled in the art will understand, the LED array may include any number of LED colors and types and combinations thereof to produce the illumination required for capturing image data according to the present invention.

[0075] The resulting image data captured at various non-overlapping spectral wavelengths can be used by object evaluation system 102 to identify one or more objects in the captured image. As those skilled in the art will understand, the wavelengths at the actual capture moment do not need to be non-overlapping, as deconvolution can be used to create images representing non-overlapping wavelengths. Image data at non-overlapping wavelengths provides measurements across the NIR wavelength spectrum sufficient to uniquely characterize most food items. Using non-overlapping wavelengths makes it more likely to generate image data that allows for better identification of one or more objects in the captured image. In other words, two overlapping wavelength spectra will result in two undesirable correlated measurements, meaning that if one measurement is large, the other measurement is also likely to be large.

[0076] Similar to the sensor fusion processing steps, additional processing steps can be employed to optimize image data captured using non-overlapping illumination wavelengths (e.g., deconvolution). According to an exemplary embodiment of the invention, image data captured at different non-overlapping illumination wavelengths can be optimized by removing ambient light using methods known in the art. For example, imaging sensor 120 can capture image data with all LEDs in the LED array off to create an environmental baseline image. The environmental baseline image can be used to convert all images captured while cycling through LEDs of specific spectral wavelengths into an image with ambient light removed (e.g., by performing subtraction). For example, image sensor 120 can capture N images (e.g., N=9), where the first image is a visible light digital image (e.g., visible light image data), and the remaining N images (e.g., 8 remaining images) are NIR digital images, each with unique LED illumination. Image sensor 120 can capture another image, the (N+1)th image, with all LEDs off. Image processing module 116 can subtract the additional (N+1)th image from the first N images (e.g., the first 9 images) to remove ambient light.

[0077] According to an exemplary embodiment of the present invention, the object evaluation system 102 may use imaging sensor 120 or imaging sensors 120a and 120b in combination with light source device 122 to capture image data required to identify one or more objects in the captured image. For example, the image processing module 116 of the object evaluation system 102 may be used to process the signals of one or more objects captured by imaging sensor 120 and perform any additional processing (e.g., sensor fusion, deconvolution, etc.). Once the image data from image sensor 120 has been captured, processed, and transformed, the object evaluation system 102 may perform further analysis on the processed and / or transformed image data. According to an exemplary embodiment of the present invention, the artificial intelligence module 118 may use the captured, processed, and transformed image data to perform further analysis for object identification. As those skilled in the art will understand, the artificial intelligence module 118 may be trained to recognize objects of any particular category or subset of categories. For example, the artificial intelligence module 118 of the object evaluation system 102 may be specifically trained to recognize food items.

[0078] According to an exemplary embodiment of the present invention, the artificial intelligence module 118 can perform a two-part analysis on captured image data from the image processing module 116 to identify one or more objects. During the first processing portion of this analysis, the artificial intelligence module 118 can run image analysis on the captured image data. The image analysis can be based on the captured visual image data, and the artificial intelligence module 118 can identify individual objects within the field of view. As those skilled in the art will understand, the field of view can include the entire region of the captured image or a subset of the entire region of the captured image. According to an exemplary embodiment of the present invention, the image analysis can analyze the visual image data to identify the unique size, shape, and color of one or more objects within the field of view. Once an object is identified as a unique object, the object evaluation system 102 can separate that individual object from the other one or more unique objects in a second processing step for further analysis. The first analysis step can continue to analyze the captured image data until all unique objects are identified and separated from the remaining objects among the one or more objects. For example, food items can be visually identified based on the analysis of unique size, shape, texture, color, etc. The unique size, texture, shape, and color of food items can provide a unique image dataset, which can be analyzed by computer algorithms based on identified pattern matching with known unique sizes, textures, shapes, and colors in a database (e.g., storage system 114). For example, Brussels sprouts can be identified by patterns matching the unique size or size range, color (specifically light green), texture, and / or shape of Brussels sprouts. According to an exemplary embodiment of the invention, visual image data can be converted into a color histogram to be used during intelligent pattern recognition or pattern matching in the artificial intelligence module 118, such as regarding... Figure 5 Further detailed discussion is needed.

[0079] According to an exemplary embodiment of the present invention, the image processing module 116 and / or the artificial intelligence module 118 can create unique spectral features for each separated object from the first processing section. In particular, the image processing module 116 and / or the artificial intelligence module 118 can use spectral image data in various spectral wavelength ranges obtained from the imaging sensor 120 to transform the captured image data to create unique spectral features for each separated object. Figure 4 Examples of various spectral ranges captured from raw spectral responses (spectral image data) are depicted. In particular, Figure 4 The true spectral reflectance of various food items (e.g., chicken, potatoes, and peas) is shown. According to an exemplary embodiment of the invention, such as... Figure 4 The depicted spectral image data can be used to create spectral features for each of one or more objects in the captured image. For example, the image data can be converted into... Figure 5 The spectral characteristics represented by the curves (e.g., using combined visible light, RGB, and NIR image data). Specifically, Figure 5 The raw image data captured by imaging sensor 120 is depicted in histogram form when targeting a diet including chicken, potatoes, and peas as food items. The image data for each food item was converted and plotted as an implicit absorption or log1 / R (R = reflectivity) response for wavelengths in nanometers. Figure 5 As shown, each plotted food item (e.g., salmon, bacon fat, bacon lean, and hamburger) has a unique response pattern (e.g., spectral features) that can be attributed to and used for identification. As those skilled in the art will understand, Figure 4 and Figure 5 The graphs depicted are for illustrative purposes only, and graphical data can be organized and stored in different forms.

[0080] According to an exemplary embodiment of the invention, a second processing portion of the analysis may utilize spectral analysis to further identify objects and / or object characteristics. Specifically, the artificial intelligence module 118 may be trained to perform spectral analysis on each segmented object and its corresponding spectral features. Spectral analysis may include performing an interaction correlation between the spectral image data of the object and previously recorded known spectral wavelengths stored in a database. As those skilled in the art will understand, the previously recorded known spectral wavelengths stored in the database may be spectral wavelengths generated by a sample of all objects programmed to be identified according to the object evaluation system 102. The artificial intelligence module 118 is trained to perform pattern matching between known spectral features in the database in use and spectral features created from the captured image data. The pattern matching of the artificial intelligence module 118 may include any known combination of algorithms for accurately predicting matching patterns. For example, the artificial intelligence module 118 may scan spectral features in the database to find the closest matching spectral feature and determine that the closest matching spectral feature is the object's identity. As those skilled in the art will understand, the artificial intelligence module 118 may identify one or more objects and / or features of those objects based on predetermined thresholds within the pattern matching.

[0081] According to an exemplary embodiment of the present invention, an artificial intelligence module 118 can be trained to recognize one or more objects. The logic of the artificial intelligence module 118 can be based on machine learning, and the AI ​​is trained to identify objects based on a combination of visible light image data and near-infrared image data. In addition to specific spectral features (e.g., color), the AI ​​module 118 can also be trained to associate the unique size, shape, and texture of one or more food items. Similarly, the AI ​​module 118 can be trained to associate specific spectral features with specific objects, and this association can be stored in a database (e.g., storage system 114). Before training the AI ​​module 118, a knowledge database must be built for one or more objects to be identified by the object evaluation system 102 in the future. For example, a user programming the object evaluation system 102 can scan all food items that are expected to be identified and create spectral features for each food item. As those skilled in the art will understand, the database can be continuously updated with new object entries created from a combination of service provider and user databases. These records of all food items, along with the names of the food items, are fed to the AI ​​training module for training the AI.

[0082] According to an exemplary embodiment of the present invention, image samples of objects can be acquired using an object evaluation system 102 or an alternative training system with a neutral background. Information about the acquired objects can be manually entered into the system, and this information will be associated with the object in a database for use by the artificial intelligence module 118. For example, the name and actual weight of the object can be recorded in TIFF and / or CSV files. Additionally, a region of interest can be selected within the image of the acquired object (limited by the object's boundaries). Multiple image planes (e.g., twenty-one planes) of the selected region of interest of the object can be obtained and stored in TIFF files. Each unique object will have a separate file (e.g., a separate CSV file) for training the artificial intelligence module 118. Once the file (e.g., CSV file) library has been created, the artificial intelligence module 118 can begin the training process. As those skilled in the art will understand, any methods and systems known in the art can be used to construct the knowledge database, and any of those methods and systems can be used to train the artificial intelligence module 118.

[0083] According to an exemplary embodiment of the invention, spectral features converted from image data and intended for use by the artificial intelligence module 118 can be represented as vectors of scalar values. These vectors of scalar values ​​can be used for pixel classification during the identification of one or more objects by the artificial intelligence module 118. For example, visible light image data may include three colors (e.g., red, green, and blue), while near-infrared image data may include fifteen colors (e.g., three colors per image captured at five specific non-overlapping wavelengths), and these colors are combined to form an eighteen-color vector for each pixel. Similarly, according to another exemplary embodiment, image data from five specific non-overlapping wavelengths can be averaged to create a single monochromatic pixel value for an eight-color vector for each pixel. Continuing with this embodiment, in the case of a vector with eighteen color values, this vector can be represented by an eighteen-dimensional hypercube (e.g., camera pixel range from 0.0 to 1.0 to create 18 possible combinations of dimensions). Since the pixels of a particular object should be relatively consistent, all points from the pixels of that object will fall close to each other in the eighteen-dimensional cube. Otherwise, pixel classification can be considered as plotting a large set of points derived from the image data to create a data point cloud. For example, a blue object may have a blue data point cloud that is clustered together within an identifiable boundary, such that when the particular blue object falls within a pre-identifiable boundary set corresponding to the particular object, the artificial intelligence module 118 can identify the presence of the particular object in the captured image. Figure 6 An example of a specific cloud, comprising numerous pixel data points, is depicted. A specific cloud can be divided into multiple segments based on color (e.g., spectral region), and boundaries can be established for the areas it falls into, associated with these colors. For example, as... Figure 6 The depicted cloud can be divided into three clouds and boundaries representing each of the red, blue, and green (RGB) regions, such that if a pixel of an object falls into the top boundary region, the object can be determined to be a red object. As those skilled in the art will understand, the artificial intelligence module 118 can use any combination of algorithms to determine the boundaries used for classifying a particular object. For example, the artificial intelligence module 118 can use K-nearest neighbor techniques, support vector machines, decision trees, Bayesian estimation, neural networks, and other methods known in the art.

[0084] Once the database has been populated with a sufficient set of spectral features, the artificial intelligence module 118 can make inferences about the image data captured by the imaging sensor 120 and identify one or more objects within the captured image data. For example, the artificial intelligence module 118 can compare data points (e.g., pixel hypercubes, boundaries, etc.) from the received image data and compare these data points with data points of known spectral features already existing in the database. Based on the comparison, the artificial intelligence module 118 can determine whether one or more objects from the captured image (e.g., the image that generated the image data) sufficiently match known objects.

[0085] According to an exemplary embodiment of the invention, the artificial intelligence module 118 can be trained to identify the unique chemical composition of an object by recognizing unique patterns associated with it using a weighted matching metric. For example, proteins (e.g., about 1050 nm) have different spectral wavelengths of reflection than carbohydrates (e.g., about 970 nm). As those skilled in the art will understand, spectral wavelengths can be selected based on the characteristic components of a material's chemical composition (e.g., according to the standard response of a known chemical substance). According to an exemplary embodiment of the invention, a second-level identification / matching utilizes the chemical characteristics of classic food components—such as fats, carbohydrates, and proteins—to define macronutrients and uses spectral color imaging to help classify food groups according to meats, vegetables, etc., where these components are more relevant to finding matches rather than determining absolute component types. For example, the artificial intelligence module 118 interprets the spectral image data of segmented sample food elements by segmenting the spectral image data into, for example, eight separate channels representing functional elements of the food. For example, foods containing hemoglobin (red meat), foods containing chlorophyll (green vegetables), foods containing fat (NIR 940 nm), foods containing carbohydrates / water (NIR 970 nm), and foods containing protein (NIR 1020 nm... detected by a 1050 nm LED). This spectral analysis of food materials is performed from visible light image data (from, for example...) Figure 5 The calculated color histogram (extracted from the image) is composed of near-infrared image data—derived from an analysis of reflectance images obtained from a specific wavelength light source (e.g., using NIR). For example, the combined visual and spectral image data can be transformed into... Figure 5 The graph shows the spectral correlation of the object. In one exemplary embodiment, Figure 5Graphical representations of visual and spectral image data captured by imaging sensor 120 are depicted when targeting a diet including salmon, bacon fat, lean bacon, and hamburgers as food items. Image data for each food item was converted and plotted as implicit absorption or log 1 / R response for wavelengths in nanometers. Figure 5 As shown, each drawn food item has a unique pattern (e.g., spectral correlation) that can be used by the artificial intelligence module 118 to identify the object.

[0086] According to an exemplary embodiment of the present invention, the object assessment system 102 can acquire additional data, and / or the additional data can be stored in a database for use in any analysis, determination, transformation, and calculation steps. The computing device 104 may include additional components for acquiring additional data via barcode scanning devices, optical character recognition, audio prompts, volatile organic compound (VOC) sensors, and manual input from the user. For example, input and output devices can use barcode scanning devices to read information from barcodes, character recognition can identify ingredient lists, microphones can capture verbal descriptions of food items spoken by the user, etc. According to an exemplary embodiment of the present invention, the object assessment system 102 can be configured to use a spectrometer in conjunction with an imaging sensor 120. The combination of the imaging sensor 120 and the light source device 122 can be used equivalently as a spectrometer by processing image data in the same way that a spectrometer processes spectral data. As those skilled in the art will understand, the imaging sensor 120 can be used for emission spectroscopy, direct absorption or reflection spectroscopy, or Raman spectroscopy. According to an exemplary embodiment of the present invention, the imaging sensor 120 may include a microelectromechanical spectrometry chip assembly, which further includes an IR source, multiple condenser lenses, a slit, an IR bandpass filter, a diffraction grating, a digital micromirror device, a detector, and a microprocessor. The imaging sensor 120 may output information related to the spectral characteristics of a food item, possibly in the form of a graph showing the absorption rate as a function of wavelength or wavenumber (reciprocal of wavelength).

[0087] According to an exemplary embodiment of the present invention, three-dimensional image data received from imaging sensor 120 can be used by image processing module 116 to determine the dimensions (e.g., surface area, volume, weight, density, etc.) of one or more objects. As those skilled in the art will understand, determining the dimensions of one or more objects can be performed independently of the identification steps discussed herein. Image processing module 116 can use three-dimensional image data to determine the volume, weight, and density of one or more objects using functions derived from visible light imaging and three-dimensional imaging. Volume, weight, and density can be obtained by image processing module 116 without requiring or having the assistance of a benchmark or reference object for scale determination. According to an exemplary embodiment of the present invention, the volume can be determined using the x, y, and z coordinates of one or more objects. According to an exemplary embodiment of the present invention, the z coordinate can be determined by determining the distance or distance length from imaging sensor 120 to the target object. The x and y coordinates can then be calculated using this distance or distance length (e.g., z coordinate). In particular, the z vector can be used to calculate the number of pixels in the image with respect to the remaining coordinates (e.g., x vector, y vector). Specifically, the imaging sensor 120 can detect the distance of pixels from the camera by using the camera's field of view as a reference, and the imaging sensor can be calibrated to provide x-coordinates, y-coordinates, and directly measure the size and shape of one or more objects. By calculating the number of pixels and obtaining the size and shape, the image processing module 116 can calculate the surface of one or more objects. Using the extent and surface area of ​​one or more objects, the image processing module can calculate other dimensions (e.g., weight, volume, etc.). According to an exemplary embodiment of the invention, the volume and weight of an object can be calculated based on visual image data.

[0088] According to an exemplary embodiment of the invention, the dimensions of each of one or more objects determined using three-dimensional image data can be used to identify the volume of different food items (or other types of objects on other surfaces) in a plated meal. As those skilled in the art will understand, determining the x, y, and z coordinates associated with a plate of food is an exemplary embodiment and is not limited to calculating the dimensions of the food items. Continuing with this exemplary embodiment, a three-dimensional scanning device 120c (or other imaging sensor 120) can be used to capture the depth or distance length between the three-dimensional scanning device 120c and the food items on the plate. The image processing module 116 can identify the bottom surface and the top surface. For example, the bottom surface can be identified as the plate or plate outline on which the food items are located, while the top surface can be a three-dimensional surface area / shape of the top surface of the food items on the plate. If the entire plate outline cannot be determined such that the shape of the plate cannot be identified, the shape of the plate can be created using the visible area of ​​the plate. In particular, the image processing module 116 can identify pixels of an image corresponding to the plate—where no food is placed on the surface to obscure the observation of the plate—and use the plate pixels to reconstruct the plate surface. For example, assuming a symmetrical disk, the system can project the visible area of ​​the disk from the center point and repeat the shape within 360 degrees to create the entire disk shape. As those skilled in the art will understand, any object can be used to represent the bottom surface associated with one or more objects being identified, and the example of a disk is not intended to be limiting. For example, continuing with the food example, a bowl, countertop, tabletop, etc., can be used as the bottom surface when determining the volume of a food item.

[0089] The volume of each food item on the plate can then be determined using the determined plate shape (or other object shape) and the three-dimensional surface area / shape of the food items. Using the plate pixels derived from three-dimensional image data from the three-dimensional scanning device 120c, the image processing module 116 can determine the three-dimensional shape of the plate (e.g., using x, y, and z coordinates) and the food items on the plate surface using the three-dimensional image data. The x, y, and z coordinates can be calculated using the identified bottom surface (e.g., the plate) and top surface (e.g., the surface area of ​​the food item). Specifically, a pixel cube can be created based on the three-dimensional image data, and the dimensions of the pixel cube (e.g., the depth between the bottom and top surfaces for the z-value, and known lateral pixel values ​​based on the distance from the sensor for the x and y values) can be used to determine the volume measurement of the pixel cube. The volume of the pixel cube can be applied to all pixels corresponding to the surface of each food item, so the volume of each food item can be determined by summing the volumes of all relevant pixels. Therefore, the identification of the bottom surface is used as a representative depth value (e.g., relative to the top surface of one or more objects) during volume calculation. The bottom surface (e.g., the disk) is not used as a reference object as in traditional methods. In other words, the established bottom surface (e.g., the disk) provides the difference (z-coordinate) between the bottom and top surfaces to be used with the x and y coordinates to calculate the volume of the object, without using a reference object as a reference for scale determination.

[0090] According to an exemplary embodiment of the present invention, volume data of a food item can be used to determine at least the weight, density, and calorie value of a particular food item. Density can be determined by utilizing the identity of the food item provided by the artificial intelligence module 118 and retrieving the density of that particular food item from a database. A specific weight of the food item can be calculated based on the volume and density. Using the weight, the object evaluation system can retrieve the weight-to-calorie conversion from the database for that particular food item.

[0091] Figure 7 and Figure 8 An exemplary flowchart depicting an implementation of the present invention is shown. Specifically, Figure 7 Depicting and showing about Figures 1 to 6 An exemplary flowchart illustrating the operation of the object evaluation system 102 under discussion. Specifically, Figure 7Process 700 is depicted, in which object assessment system 102 captures an image of a diet and provides nutritional information about the food items in the diet. In step 702, the process is initiated by a user triggering imaging sensor 120 over a plate of food. For example, the user swings computing device 104 over the plate of food and initiates object assessment system 102. The swinging of computing device 104 over the plate of food can trigger motion-sensing actuator 304 to transmit a signal to enable imaging sensor 120 to capture image data. Once the image capture process has been initiated, the user should hold computing device 104 in a fixed position to capture image data clearly and accurately. According to an exemplary embodiment of the invention, computing device 104 may display an indication that the capture process has been initiated, instructing the user to keep computing device 104 stationary over the target object until imaging sensor 120 completes capturing image data. As those skilled in the art will understand, imaging sensor 120 may be activated by another device (e.g., an actuator / button implementation of actuator 304).

[0092] In step 704, imaging sensor 120 detects a plate setting on the table and triggers imaging sensor 120 to capture image data of the food on the plate. For example, imaging sensor 120 uses three-dimensional image data to detect the edge of the plate based on the depth difference between the plate and the table surface at the edge of the plate. As those skilled in the art will understand, imaging sensor 120 can identify one or more other objects within the field of view, not just the plate. In step 704, image processing module 116 can also determine the distance length or distance between imaging sensor 120 and the plate. According to an exemplary embodiment of the invention, image processing module 116 can identify the plate by recognizing predetermined plate primitives from a set of plate primitives stored in storage system 114. When a plate is detected, imaging sensor 120 is instructed to capture image data of the plate. As those skilled in the art will understand, if multiple plate settings exist, object evaluation system 102 can capture image data of each plate setting or a single central plate setting.

[0093] In step 706, the imaging sensor 120 captures visible light image data, spectral image data, and three-dimensional image data, and the image processing module 116 stores the captured image data for further processing. The visible light image data, spectral image data, and three-dimensional image data of the entire food item are automatically obtained, as shown in the image above. Figures 1 to 6As discussed. For example, image sensor 120 can capture visible light image data in a single image capture (e.g., using RGB sensor 120b) and spectral image data (e.g., using NIR sensor 120a) by capturing a sequence of images simultaneously triggered for a unique illumination wavelength for each desired NIR (e.g., using LED array 122). In step 708, image processing module 116 and artificial intelligence module 118 process each individual image and image set, as discussed in... Figures 1 to 6 As discussed, for example, visual image data is used by the artificial intelligence module 118 to identify and separate individual food items based on their unique size, shape, texture, and color using pattern matching. Similarly, spectral image data is used to determine the unique spectral characteristics of each individual food item by performing cross-matching by the artificial intelligence module 118.

[0094] In step 710, the image processing module 116 can identify pixels in the image corresponding to the plate—where no food is placed on the surface to obscure observation of the plate. Using the plate pixels, the image processing module 116 can use the 3D scanning device 120c to determine the 3D shape of the plate (e.g., using x, y, and z coordinates). According to an exemplary embodiment of the invention, if the entire plate outline cannot be determined to the point that the shape of the plate cannot be identified, the shape of the plate can be created using the visible area of ​​the plate. For example, assuming a symmetrical plate, the system can project the visible area of ​​the plate at the center point and rotate the shape 360 ​​degrees to create the plate shape. The determined plate shape and the 3D surface area / shape of the food items can then be used to determine the volume of each food item on the plate. In step 712, using the identification in step 708 and the pixel size determination in step 710, the image processing module 116 can calculate the volume of the food items. For example, the image processing module 116 can calculate the number of pixels in each identified food item and then use the number of pixels to determine the volume. Figure 7 As a result of process 700, object evaluation system 102 can use computing device 104 to capture image data of a plate of food, identify food items on the plate, and determine the volume of each food item.

[0095] Figure 8 Describes the use of from Figure 7 ( Figures 1 to 6 The process 800 involves determining the nutritional value of a food item based on its identification and volume information. In step 802, the image processing module 116 can acquire an initial list of food items and their corresponding volumes. For example, the image processing module 116 can determine the nutritional value of a food item based on its identification information, such as the nutritional value of the food item in the image processing module. Figure 7The image processing module 116 obtains a list of food items and their corresponding volumes based on the information identified in the previous steps. In step 804, the image processing module 116 can obtain the final list of food items and their corresponding volumes. For example, similar to step 802, the image processing module 116 can obtain the final list of food items and their corresponding volumes based on the information identified in the previous steps. Figure 7 The steps involve obtaining a list of food items and their corresponding volumes. In step 806, the image processing module 116 can calculate the volume change from the initial volume to the final volume. For example, the image processing module 116 can use subtraction to calculate the difference between the initial volume of the food item in step 802 and the final volume of the food item in step 804. In step 808, the image processing module 116 can connect to a database to retrieve dietary information for the listed food items. For example, the image processing module 116 can contact the FDA and / or USDA databases and request nutritional information from the food item list in step 802. In step 810, the image processing module 116 can calculate the dietary content of the food items based on the calculated volume change. For example, the image processing module 116 can use nutritional information obtained from the FDA and / or USDA databases and calculate the dietary content of the food items based on the volume change calculated in step 806.

[0096] According to an exemplary embodiment of the present invention, an end user can use the object assessment system 102 to perform tasks related to the identification of one or more objects, the determination of the volume of one or more objects at different points in time, and the collection of additional information about the one or more objects based on the identification and determination steps. For example, an end user can use the object assessment system 102 as part of a diet tracking application to automatically identify food items and the nutritional information associated with those food items. As those skilled in the art will understand, the object assessment system 102 can be used to identify any combination of identifiable objects and perform further analysis thereon.

[0097] The following embodiments are exemplary models of a specific application of the object evaluation system 102 in identifying food items. The invention is not intended to be limited to the identification of food items, and these embodiments are for illustrative purposes only. Following prompts from a diet tracking application, the end user will use the computing device 104 to capture image data of the diet before any consumption occurs. According to an exemplary embodiment, the end user can activate the object evaluation system 102 by triggering an actuator 304 on the computing device 104. The actuator 304 can be activated by “swinging” the computing device 104 over the plate containing the food to be consumed. Figure 9 Examples of diet can be seen in the text. Specifically, Figure 9A diet consisting of three individual food items—peas, chicken, and mashed potatoes—is depicted. The "swing" triggers an automated process that begins by automatically instructing the imaging sensor 120 to identify the plate / plate shape, depth, and capture visual, spectral, and 3D image data of the entire diet. All image data is automatically acquired by the image sensor 120 and transmitted to the image processing module 116 and the artificial intelligence module 118 for further processing. (See also: [link to related information]) Figures 1 to 8 As discussed, the image processing module 116 and the artificial intelligence module 118 can use the captured image data to separate individual food items (e.g., peas, chicken, mashed potatoes), identify the separated food items, calculate the volume of the food items (e.g., using three-dimensional image data), and determine the nutritional value of the food items in the initial diet. As those skilled in the art will understand, nutritional values ​​can be retrieved from a database by looking up the nutritional values ​​of the identified food items, and the nutritional value can be calculated based on the determined volume (partial size) of the food item. According to an exemplary embodiment of the invention, the object assessment system 102 can contact remote databases (e.g., FDA and / or USDA databases) to collect nutritional values ​​of food items.

[0098] As those skilled in the art will understand, the end user can add additional information about the diet used during the analysis. For example, the end user can use a microphone to capture verbal descriptions. Any additional information provided by the user can be used for further analysis of food items. For example, the end user can indicate whether the chicken was grilled or baked, and the object evaluation system 102 can take the preparation type of the chicken into account when determining nutritional values. As those skilled in the art will understand, these processing steps can be performed simultaneously or sequentially. Once the user has consumed a full serving or a portion of the diet, the actuation mechanism can be used again by “swinging” the computing device 104 over the remaining uneaten food on the plate. The automated processing steps of the image processing module 116 and the artificial intelligence module 118 are repeated, and the food consumption data stored by the web-based application is revised to take into account the food items not eaten by the user. The diet application will present the resulting nutritional information to the user based on the amount of food consumed by the user (e.g., the volume difference from the initially captured image data to the final captured image data).

[0099] According to an exemplary embodiment of the present invention, nutritional information can be presented to the end user (e.g., on a diet tracking application on a user's mobile computing device). Figure 10A and Figure 10B Depicting Figure 9An exemplary illustration of the nutritional values ​​of the depicted diet. Specifically, the present invention captures an image of the entire diet in a single capture of image data of the entire plate within the field of view using image sensor 120, and immediately generates a nutritional food label for the entire diet (e.g., comprehensive nutritional information for all food items). Without requiring another image capture, the user can scroll through the segmented or automatically separated nutritional food labels of the individual food items constituting the entire diet on computing devices 104, 124, such as... Figure 10B As shown. Individual nutrition labels may include the actual weight of each food item. Specifically, as... Figure 10B As shown, Figure 9 Each individual food item in the diet (e.g., peas, chicken, mashed potatoes) can have a separate nutrition label showing all the nutritional values ​​of the consumed food item. According to an exemplary embodiment of the invention, nutritional food labels can be created based on the volume of food consumed by the user (e.g., using images before and after capture). As those skilled in the art will understand, any combination of data related to the user's health and / or diet can be presented to the user. For example, a diabetic user can be notified that they have exceeded their daily carbohydrate / sugar / other nutrient limits and the ratio of fat to carbohydrates to sodium, etc. Similarly, the system can take into account data on the user's physical activity before and after meals, recorded in the user's account. For example, if the user exercised before a meal, the system can instruct the user to enjoy extra calories during the next meal.

[0100] According to an exemplary embodiment of the present invention, the diet application can be a web-based application that receives, processes, synthesizes, and displays diet information based on calculations performed using image data sent from computing device 104 combined with data retrieved from multiple public and private databases (e.g., system memory 114). The web-based application helps users collect, track, and record information related to their diet. The web-based application may also include user accounts, a graphical user interface, and an application programming interface suite. When a user begins using the object assessment system 102 and computing device 104, the user first establishes a user account linked to one or more of the user's devices (e.g., other computing devices 124). When creating a user account, the user enters information, including but not limited to data related to the user's healthy eating needs and goals, as well as food aversions and allergies. The user account automatically stores, processes, and organizes food consumption data collected by the user's one or more devices. The graphical user interface enables users to access, view, enter, and modify information related to their user account.

[0101] According to an exemplary embodiment of the present invention, the dietary application may include an application programming interface (API) suite that utilizes data from multiple databases combined with information collected by computing device 104 to facilitate the functionality of object evaluation system 102. The API suite may include an imaging sensor application programming interface (API), a 3D scanning device API, a visual comparison API, a voice recognition API, an automatic segmentation API, a volume-to-weight API, a nutritional value output API, a 3D printed model API, a barcode and optical character recognition API, and a food inventory API. The imaging sensor API may use image data related to the spectral characteristics of food items output by imaging sensor 120 of computing device 104, combined with multiple chemometric algorithms from a chemometric algorithm database, to identify those foods.

[0102] Similarly, the 3D scanning device API can convert 3D image data (e.g., point cloud data) in .asc or .txt files output by the 3D scanning device of computing device 104 into files that can be read by computer-aided design software, including but not limited to .stl or .ply files. The visual comparison API can use 2D and / or 3D image data captured by the imaging sensor 120 of computing device 104 combined with multiple food image databases (e.g., storage system 114) to identify food through 2D or 3D images, such as... Figures 1 to 9 The discussed speech recognition API can analyze audio information captured by the microphone of computing device 104 to identify the food described by the user. According to an exemplary embodiment of the invention, both the visual comparison API and the speech recognition API are used when the imaging sensor API cannot identify the food on the plate.

[0103] An automatic segmentation API utilizing CAD files output from a 3D scanning device API can be combined with a proprietary automatic segmentation algorithm of the object evaluation system 102, spectral feature recognition from an imaging sensor API, and / or 2D or 3D image data to automatically segment a dish containing several different foods into the various constituent foods of that dish, and calculate the volume and dimensions of each constituent food. The volume-to-weight API uses food identification data from a stoichiometric algorithm, data from the 3D scanning device, and 2D or 3D camera images, and cross-references this data with databases—including USDA databases, FDA and / or USDA databases, and online databases—to derive the food density, specific weight calculation constant, and specific calorie calculation constant for those foods. The food density, specific weight calculation constant, and specific calorie calculation constant are used in conjunction with the volume of those foods calculated by the automatic segmentation API to calculate the weight and calorie content of the scanned food. The weight and calorie content of the food are then stored via a web-based application.

[0104] The Nutrition Values ​​Output API can use food identification data and 2D or 3D image data from chemometric algorithms to cross-reference the image data with data stored in databases—including USDA databases, FDA and / or USDA databases and online databases—to derive nutritional information for the diet or individual dietary components, including but not limited to calories, protein, total fat, saturated fat, and fatty acids. This information is then displayed on a graphical user interface in a format similar to nutrition facts labels found on many food packages. The 3D Printing Model API allows users to export .stl or .ply CAD files output by the 3D Scanning API to 3D printing modeling software to create 3D printed models of the diet. These 3D printed models can be used by individual users as models with ideal serving sizes for portion control, or by larger canteens or industrial food operations to ensure sufficient food production and design ideal packaging for the food. The Barcode and Optical Character Recognition API can use algorithms to analyze images of Universal Product Codes (UPCs) or ingredient lists captured by a camera on computing device 104 to automatically import the food's nutritional information into the Nutrition Values ​​API and identify and warn users of potential allergens or food aversion conflicts. For example, users in grocery stores can use barcode and optical character recognition APIs to quickly and easily scan packaged items on shelves to check for allergens or unhealthy ingredients.

[0105] The Food Inventory API allows users to automatically track the amount of food stored. To use the Food Inventory API, users first scan ingredients when storing them, such as when an item is first brought home from the grocery store and placed in the refrigerator or pantry. This establishes an inventory baseline. Then, once a meal is prepared, the meal is scanned, and the ingredients in that meal are subtracted from the inventory baseline. The Food Inventory API can generate a shopping list with the items and associated quantities needed to restore the inventory baseline. The Food Inventory API works with GPS beacons to ensure that only the ingredients of meals consumed near the inventory are subtracted from the inventory baseline. For example, if a user brings home 16 ounces of chicken from the grocery store, then scans 8 ounces of chicken at home after preparing a home-cooked meal, and then scans 8 ounces of chicken at a restaurant while dining out, only the 8 ounces of chicken scanned at home will be subtracted from the chicken's inventory baseline. Restaurants or canteens can also use the Food Inventory API to manage their inventory.

[0106] Any suitable computing device can be used to implement computing devices 102, 104 and the methods / functions described herein, and it can be converted into a specific system by modifying the hardware, software and firmware to perform the operations and features described herein in a manner that is significantly different from simply executing software on a general-purpose computing device, as will be understood by those skilled in the art. Figure 11 An illustrative embodiment of such a computing device 1600 is described herein. The computing device 1600 is merely an illustrative embodiment of a suitable computing environment and does not limit the scope of the invention. Figure 11 The term "computing device" as illustrated may include a "workstation," "server," "laptop computer," "desktop computer," "handheld device," "mobile device," "tablet computer," or other computing device, as will be understood by those skilled in the art. Considering that the computing device 1600 is depicted for illustrative purposes, embodiments of the present invention may utilize any number of computing devices 1600 in various different ways to implement a single embodiment of the invention. Therefore, as will be understood by those skilled in the art, embodiments of the present invention are not limited to a single computing device 1600, nor are they limited to a single type of implementation or configuration of the exemplary computing device 1600.

[0107] Computing device 1600 may include a bus 1610 that may be directly or indirectly coupled to one or more of the following illustrative components: memory 1612, one or more processors 1614, one or more presentation units 1616, input / output ports 1618, input / output units 1620, and power supply 1624. Those skilled in the art will understand that bus 1610 may include one or more buses, such as an address bus, a data bus, or any combination thereof. Those skilled in the art will also understand that, depending on the intended application and use of a particular embodiment, multiple components may be implemented by a single device. Similarly, in some cases, a single component may be implemented by multiple devices. Therefore, Figure 11 This description merely illustrates exemplary computing devices that can be used to implement one or more embodiments of the present invention and does not limit the invention.

[0108] The computing device 1600 may include or interact with a variety of computer-readable media. For example, computer-readable media may include: random access memory (RAM); read-only memory (ROM); electrically erasable programmable read-only memory (EEPROM); flash memory or other memory technologies; CD-ROM, digital universal disc (DVD) or other optical or holographic media; magnetic tape cassettes, magnetic tape, disk storage devices, or other magnetic storage devices that can be used to encode information and can be accessed by the computing device 1600.

[0109] Memory 1612 may include computer storage media in the form of volatile and / or non-volatile memory. Memory 1612 may be removable, non-removable, or any combination thereof. Exemplary hardware devices are devices such as hard disk drives, solid-state storage, optical disk drives, etc. Computing device 1600 may include one or more processors that read data from components such as memory 1612, various I / O components 1620, etc. Presentation component 1616 presents data indications to a user or other device. Exemplary presentation components include display devices, speakers, printing components, vibrating components, etc.

[0110] I / O port 1618 allows computing device 1600 to be logically coupled to other devices such as I / O components 1620. Some of the I / O components 1620 can be embedded in computing device 1600. Examples of such I / O components 1620 include microphones, joysticks, recording devices, game controllers, satellite antennas, scanning devices, printers, wireless devices, network devices, etc.

[0111] As used herein, the terms “comprise” and “comprising” are intended to be interpreted as inclusive, not exclusive. As used herein, the terms “illustrative,” “exemplary,” and “illustrative” are intended to mean “serving as an example, instance, or illustration” and should not be construed as indicating or not indicating a preferred or advantageous configuration relative to other configurations. As used herein, the terms “about” and “approximately” are intended to cover possible variations in the upper and lower limits of a range of subjective or objective values, such as variations in properties, parameters, sizes, and dimensions. In a non-limiting embodiment, the terms “about” and “approximately” mean close to, or plus 10% or less, or minus 10% or less. In a non-limiting embodiment, the terms “about” and “approximately” mean a value that a person skilled in the art would consider sufficiently close to be included in the relevant field. As used herein, the term “substantially” refers to a complete or nearly complete extension or extent of an action, feature, characteristic, state, structure, item, or result, as would be understood by a person skilled in the art. For example, a "substantially" circular object would mean that the object is perfectly circular to a mathematically determinable degree, or close to being circular as would be recognized or understood by a person skilled in the art. In some cases, the exact permissible deviation from absolute completeness may depend on the specific circumstances. However, in general, a degree of closeness to completeness will achieve the same overall effect as absolute and total completeness achieved or obtained. As a person skilled in the art will understand, when "substantially" is used in the negative sense, it is equally applicable to indicating a complete or near-complete lack of an action, feature, property, state, structure, item, or result.

[0112] Based on the foregoing description, many modifications and alternative embodiments of the invention will be apparent to those skilled in the art. Therefore, this specification is to be understood as illustrative only and for the purpose of teaching those skilled in the art the best mode for carrying out the invention. Structural details may vary substantially without departing from the spirit of the invention, and exclusive use of all modifications falling within the scope of the appended claims is reserved. Embodiments have been described in this specification in a manner intended to make the description clear and concise; however, it is intended and understood that embodiments may be combined or separated differently without departing from the invention. The invention is intended to be limited only to the scope and applicable provisions claimed by the appended claims.

[0113] It should also be understood that the appended claims will cover all the general and specific features of the invention described herein, and all statements of the scope of the invention in terms of language are deemed to fall within it.

Claims

1. A system for object identification and evaluation, comprising: Digital cameras, including: At least one image sensor, the digital camera being configured to convert visible light captured within a spectral range and near-infrared (NIR) light captured within a spectral range into a captured photovoltage; and At least one image processing module that converts the captured photovoltage into three-dimensional (3D) image data; and A recording device that records the 3D image data; The digital camera captures and records at least two distinct and non-overlapping subsets of spectral ranges within the NIR spectral range, without recording the spectral range gaps between the non-overlapping subsets of spectral ranges. The image processing engine is configured as follows: The captured and recorded photovoltages are analyzed by creating spectral features from the captured photovoltages to identify one or more objects, wherein the spectral features include a vector of scalar values ​​representing the spectral response across the visible light spectrum and non-overlapping NIR spectral wavelengths; and Based on the recorded 3D image data, determine the volume data of the one or more objects over a given time period; and The artificial intelligence module is configured as follows: The one or more objects are identified by matching the spectral features with a database of known spectral features corresponding to known objects; Based on the identification, feature information data of one or more identified objects are obtained from one or more databases; and A display device for outputting volume data and feature information data of the one or more objects.

2. The system according to claim 1, wherein: The visible light is captured in the spectral range of 400 nm to 700 nm; and The NIR light is captured in the spectral range of 700 nm to 1050 nm.

3. The system according to claim 1, wherein, The at least one image sensor further includes: A 3D scanning device configured to capture 3D point cloud data of the one or more objects; A digital NIR camera with a visible light blocking filter is configured to capture the photovoltage of the one or more objects; and An RGB digital camera configured to capture two-dimensional or three-dimensional image data of the one or more objects.

4. The system according to claim 3, wherein, The digital NIR camera captures a series of two or more images of the at least two distinct and non-overlapping subsets of spectral ranges by passing through the NIRLED array one at a time in a loop.

5. The system according to claim 1, wherein, The captured photovoltage is used to identify one or more sub-objects of the one or more objects.

6. The system according to claim 5, wherein, Separate one or more identified sub-objects from the one or more objects, and obtain feature information data for each of the one or more sub-objects from the one or more databases.

7. The system according to claim 3, wherein, The artificial intelligence module identifies at least one predefined primitive and triggers the RGB digital camera and the digital NIR camera to align with the at least one predefined primitive.

8. The system according to claim 1, wherein, The at least one image processing module is configured to use three-dimensional point cloud data of the one or more objects to determine the distance length from the at least one image sensor to the one or more objects, determine the surface area of ​​the one or more objects, and calculate the size and shape of the one or more objects using time of flight and the surface area without the need to use a reference marker.

9. The system according to claim 8, wherein, The flight time and surface area of ​​the one or more objects are determined by calculating the number of pixels in the three-dimensional point cloud data within the field of view.

Images