Method and system for processing images

Abstract

A method of processing an image, the method comprising: receiving image data from a subject at a remote location; and analyzing the image data to identify therein content related to the food category. A set of subject descriptor features specific to the subject is obtained, and a trained machine learning program is applied to the identified food category and the set of features to provide a score describing a response of the subject to the food category. The method generates processed image data including a score superimposed on at least a portion of the received image data, and transmits the image data to a display device viewable by the subject at the remote location.

Description

[0001] Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 459,936, filed April 17, 2023, the contents of which are incorporated by reference in their entirety. BACKGROUND

[0002] In some embodiments of the invention, the invention relates to image processing, and more particularly, but not exclusively, to a method and system for processing images to provide an individual with a personalized diet.

[0003] Computerized dieting techniques have been disclosed, for example, International Publication No. WO 2002100266 discloses a dieting technique employing a database of reference human factors. A computer receives user data regarding these human factors and, from the user data and measured blood properties, predicts selected properties of the user's blood to generate a prediction model. The prediction model is then queried to generate and display predicted blood properties dependent on the input human factor data. Use of a database comprising prediction models of selected blood properties as a function of human factors is also disclosed.

[0004] International Publication No. WO2015 / 166489, the contents of which are hereby incorporated by reference, discloses a method of predicting a subject's response to food by selecting food for which the subject's response is unknown and using information describing the subject and knowledge of other subjects' responses to food to estimate the subject's response to the selected food. SUMMARY

[0005] According to an aspect of some embodiments of the invention, there is provided a method of processing an image. The method comprises receiving image data from a subject at a remote location and analyzing the image data to identify content therein relating to a food category. The method further comprises obtaining a set of subject descriptor features specific to the subject and applying a trained machine learning procedure to the identified food category and the set of features to provide a score describing the subject's response to the food category. The method generates processed image data containing the score superimposed on at least a portion of the received image data and transmits the image data to a display device viewable by the subject at the remote location.

[0006] According to some embodiments of the invention, the image data is analyzed by identifying non-textual content in the image.

[0007] According to some embodiments of the invention, the image data is analyzed by applying character recognition to textual content in the image.

[0008] According to some embodiments of the application, the image data comprises a stream of image data, and the processed image data comprises a stream of processed image data.

[0009] According to some embodiments of the application, the processed image data is also generated by superimposing identification information of the food items.

[0010] According to some embodiments of the application, the image data is analyzed by identifying different content related to different food items in the image data, wherein the application of the trained machine learning program comprises applying the trained machine learning program individually to each of the food items to provide a score describing the subject's reaction to each of the food items, and wherein the processed image data is generated by superimposing each of the scores on at least a portion of the received image.

[0011] According to some embodiments of the application, at least two of the different content are in a single image frame.

[0012] According to some embodiments of the application, at least two of the different content are in different image frames.

[0013] According to some embodiments, the method comprises calculating a meal score for the selected food items, and transmitting the meal score to the display device.

[0014] According to some embodiments of the application, the selection comprises all food items.

[0015] According to some embodiments of the application, the selection comprises a portion of the food items. According to some embodiments of the application, the portion is automatically selected. According to some embodiments, the method comprises receiving the portion from the subject at the remote location.

[0016] According to an aspect of some embodiments of the present application, there is provided a computer software product. The computer software product comprises a computer-readable medium having stored therein program instructions that, when read by a data processor, cause the data processor to carry out the method as described above and, optionally and preferably, as further detailed below.

[0017] According to an aspect of some embodiments of the present application, there is provided a server system for processing images. The server system comprises a transceiver arranged to receive and transmit image data over a communication network, and a processor arranged to communicate with the transceiver and carry out the method as described above and, optionally and preferably, as further detailed below.

[0018] According to an aspect of some embodiments of the present application, there is provided a system for processing images. The system comprises an imaging device for capturing image data from a scene, a display device, and a data processor. The data processor is configured to transmit the image data to a server, receive processed image data from the server, wherein the captured image data is overlaid with scores describing a subject's reaction to a food item or a representation of a food item contained in the captured image data, and display a graphical user interface (GUI) containing an image corresponding to the processed image data on the display device.

[0019] According to some embodiments of the present application, the data processor is configured to receive a selection of the food item from the GUI, add the selection to a list of food items defining a meal, transmit the list to the server, responsively receive a meal score of the list from the server, and display the meal score on the GUI.

[0020] According to an aspect of some embodiments of the present application, there is provided a method of displaying processed image content. The method comprises capturing image data from a scene containing a food item or a representation of a food item, transmitting the image data to a server, receiving processed image data from the server, wherein the captured image data is overlaid with scores describing a subject's reaction to the food item, and displaying a graphical user interface (GUI) containing an image corresponding to the processed image data on a display device.

[0021] According to some embodiments of the present application, the capturing of the image data comprises capturing an image of a food item.

[0022] According to some embodiments of the present application, the capturing of the image data comprises capturing an image of a food menu containing textual content describing a plurality of food items.

[0023] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application pertains. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of embodiments of the application, exemplary methods and / or materials are described below. In case of conflict, the patent specification, including definitions, will control. In addition, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting.

[0024] Implementation of the method and / or system of embodiments of the application can involve performing or completing selected tasks manually, automatically, or a combination thereof. Moreover, according to actual instrumentation and equipment of embodiments of the method and / or system of the application, several selected tasks could be implemented by hardware, by software or by firmware or by a combination thereof.

[0025] For example, hardware for performing selected tasks according to embodiments of the application could be implemented as a chip or a circuit. As software, selected tasks according to embodiments of the application could be implemented as a plurality of software instructions being executed by a computer using any suitable operating system. In an exemplary embodiment of the application, one or more tasks according to exemplary embodiments of method and / or system as described herein are performed by a data processor, such as a computing platform for executing a plurality of instructions. Optionally, the data processor includes a volatile memory for storing instructions and / or data and / or a non-volatile storage, such as a magnetic hard-disk and / or removable media. Optionally, the data processor includes a display for displaying information and / or an alphanumeric input device, such as a keyboard, for inputting information. Optionally, the data processor further includes a user input device, such as a touch screen or a pointing device, for providing interactive input to the computer. The user input device is used for inputting data to the computer and communicating user commands to the computer. The user input device optionally includes an input device, such as a touchscreen or a pointing device, for communicating user input information and commands to an electronic system. The user input device optionally includes a display, such as a touchscreen, for displaying video and / or graphics. BRIEF DESCRIPTION OF DRAWINGS

[0026] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0027] Some embodiments of the application will now be described, by way of example only, with reference to the attached figures. It is to be understood that the details set forth herein are by way of example and are not intended to limit the scope of the application, as defined in the appended claims. In this respect, before explaining several embodiments of the application in details, it is to be understood that the application is not limited in its application to the details set forth in the following description or exemplified by the Examples. The application is capable of other embodiments or of being practiced or carried out in various ways.

[0028] In the drawings: Figure 1 is a flowchart of a method suitable for processing images according to some embodiments of the application; Figure 2 is a schematic illustration of a client-server configuration that can be used for providing a personalized diet to a subject according to some embodiments of the application; and Figures 3A to 3E A graphical user interface (GUI) that can be used according to some embodiments of the application is shown. DETAILED DESCRIPTION

[0029] In some embodiments of the application, the present application relates to personalized diets, and more particularly, but not exclusively, to a method and system for improving the diet of an individual.

[0030] Before one or more embodiments of the application are explained in detail, it is to be understood that the application is not limited in its application to the details of construction and the arrangement of the components and / or methods set forth in the following description and / or illustrated in the drawings and / or the Examples. The application is capable of other embodiments or of being practiced or carried out in various ways.

[0031] Reference will now be made to the drawings, Figure 1 is a flowchart of a method for processing images according to various exemplary embodiments of the present application. It should be understood that the operations described below can be performed in many execution combinations or orders, either simultaneously or sequentially, unless otherwise defined. In particular, the ordering of the flowcharts should not be considered limiting. For example, two or more operations appearing in a particular order in the following description or flowcharts can be performed in a different order (e.g., reverse order) or substantially simultaneously. In addition, several of the operations described below are optional and can not be performed.

[0032] At least a portion of the operations described herein can be implemented by a data processing system (e.g., special-purpose circuitry or a general-purpose processor) configured to perform the operations described below. At least a portion of the operations can be implemented by a cloud computing facility at a remote location.

[0033] A computer program implementing the method of the present embodiment can typically be distributed to users on a communication network or a distribution medium (such as, but not limited to, a floppy disk, a CD-ROM, a flash memory device, and a portable hard disk drive). The computer program can be copied from the communication network or the distribution medium to a hard disk or similar intermediate storage medium. The computer program can be run by loading the code instructions from its distribution medium or intermediate storage medium into the execution memory of a computer, thereby configuring the computer to action in accordance with the method of the present application. During operation, the computer can store data structures or values in the memory and retrieve them for subsequent operations. All these operations are well-known to those skilled in the art of computer systems.

[0034] The processing operations described herein can be performed by means of a processor circuit, such as a DSP, a microcontroller, an FPGA, an ASIC, etc., or any other conventional and / or specialized computing system.

[0035] The method of the present embodiment can be embodied in a variety of forms. For example, it can be embodied on a tangible medium, such as a computer for performing the method operations. It can be embodied on a computer-readable medium, which includes computer-readable instructions for performing the method operations. It can also be embodied in an electronic device having digital computer capabilities, arranged to run a computer program on a tangible medium or to execute instructions on a computer-readable medium.

[0036] The method of this embodiment preferably processes image data for the purpose of providing a personalized diet to a subject. In various exemplary embodiments of the invention, the method is performed on behalf of a service provider, such as but not limited to a provider of personalized diet recommendations. In these embodiments of the invention, the subject is a member of a group of identified subjects, each subject has an existing user account with the service provider, and the subject is requested to log into their user account to allow the method to provide services to the identified subject.

[0037] The method begins at 10, and optionally and preferably continues to 11, where image data is received from a subject at a remote location.

[0038] When the subject has an existing user account with the service provider, the method can identify the subject based on the user account and associate the received image data with the identified subject.

[0039] The image data can be captured by an imaging device at the remote location, and transmitted to the server system for processing as described below. For example, an app of a mobile device (e.g., a smartphone or tablet computer, etc.) can have permission (e.g., a user-enabled permission) to use the camera of the mobile device, and the subject can operate the camera-enabled app to capture image data from a scene. The image data can be in the form of a single still image, or a set of still images, or video images. A single still image is equivalent to a single frame of a video image, and is therefore referred to as an image frame hereinafter. The image data transmitted to the server system can also be a stream of image data, e.g., a sequence of image frames.

[0040] The imaged scene can include one or more physical food items, or a representation of one or more food items, such as but not limited to an image or sketch or artificial object having the shape and optionally color of the food item, or textual content describing one or more food items, such as but not limited to a menu of food items.

[0041] The method continues to 12, where a set of subject descriptor features specific to the subject is obtained. Typically, these descriptor features are read from a computer-readable medium that stores a set of descriptor features for each of a plurality of identified subjects having a user account with the service provider. Alternatively, the descriptor features can be uploaded from the remote location as a data file associated with the received meal.

[0042] Descriptor features typically include general information such as gender and age, and preferably also the subject's population. This population is typically a demographic group and indicates the subject's demographic characteristics, including but not limited to the subject's geographic origin, current geographic area, and cultural customs. Descriptor features may additionally include at least one of the following: the subject's microbiome profile or a portion thereof obtained by analyzing a fecal sample; the subject's blood chemistry or a portion thereof [e.g., the results of one or more tests, such as, but not limited to, triglyceride testing, glycosylated hemoglobin (HbA1c) testing, C-peptide testing, fasting plasma glucose (FPG) testing, oral glucose tolerance test (OGTT), and random plasma glucose testing]; the subject's genetic profile or a portion thereof; metabolomics data associated with the subject; and the subject's medical condition. Descriptor features may optionally and preferably also include body type parameters such as height, weight, and / or BMI. In the inventors’ experiments, the following descriptive features were used: population, gender, age, height, weight, BMI, FPG test results, triglyceride test results, HbA1c test results, and microbiome-related features extracted from fecal samples of the subjects.

[0043] The method continues to step 13, where image data is analyzed to identify content related to food categories. This can be accomplished using any known image recognition technique, such as, but not limited to, one of the image recognition methods disclosed in U.S. Patent Nos. 10,860,930, 11,164,294, and 11,270,169, or one of the commercial software products available from Kooaba Corporation or Idee, Inc.'s TiiiEye, Fiximilar, or PixMatch API. The analysis at step 13 may include segmenting the image frame into multiple tiles and analyzing the image data corresponding to each tile individually. In some embodiments of the invention, the method identifies different content related to different food categories in the image data. Different food categories may be contained within a single image frame (e.g., in different tiles of the image frame) and / or contained in different image frames.

[0044] The method optionally and preferably continues to step 14, wherein the method predicts the subject's response to a food category. The prediction uses a set of subject descriptor features obtained at step 12, and is therefore specific to the subject transmitting image data to the server. Preferably, the prediction is represented by a numerical score indicating the degree of the subject's response to the food category. Typically, the predicted response is the subject's blood glucose response to the food category, but other types of responses are also considered, such as, but not limited to, changes in cholesterol, sodium, potassium, and / or calcium levels. Thus, the score can indicate the predicted glucose level and / or predicted cholesterol level, and / or predicted sodium level, and / or predicted potassium level, and / or predicted calcium level in the subject's blood after consuming the food category.

[0045] The prediction can be made, and preferably, by means of a machine learning program trained to predict responses to food categories based on subject descriptor features.

[0046] In machine learning, information can be acquired through supervised or unsupervised learning. In some embodiments of the present invention, the machine learning program includes or is a supervised learning program. In supervised learning, a global or local objective function is used to optimize the structure of the learning system. In other words, in supervised learning, there is a desired response that the system uses to guide the learning process.

[0047] In some embodiments of the invention, the machine learning program includes or is an unsupervised learning program. In unsupervised learning, there is typically no objective function. Specifically, the learning system does not set a set of rules. According to some embodiments of the invention, one form of unsupervised learning is unsupervised clustering, where the data objects do not have... A priori Category tags.

[0048] Representative examples of "machine learning" programs applicable to this embodiment include, but are not limited to, clustering, association rule algorithms, feature evaluation algorithms, subset selection algorithms, support vector machines, classification rules, cost-sensitive classifiers, voting algorithms, stacking algorithms, Bayesian networks, decision trees, neural networks, instance-based algorithms, linear modeling algorithms, k-nearest neighbor analysis, ensemble learning algorithms, probabilistic models, graphical models, regression methods, gradient ascent methods, singular value decomposition methods, and principal component analysis. In some embodiments of the present invention, the machine learning program is a program that employs decision trees.

[0049] The following is an overview of some machine learning procedures applicable to this embodiment.

[0050] Support Vector Machines (SVMs) are algorithms based on statistical learning theory. According to some embodiments of the present invention, SVMs can be used for classification purposes and / or for numerical prediction. SVMs used for classification are referred to herein as "support vector classifiers," and SVMs used for numerical prediction are referred to herein as "support vector regressions."

[0051] SVMs are typically characterized by kernel functions, the choice of which determines whether the resulting SVM provides classification, regression, or other functions. By applying a kernel function, an SVM maps the input vectors to a high-dimensional feature space, in which a decision hypersurface (also called a separator) can be constructed to provide classification, regression, or other decision functions. In the simplest case, the surface is a hyperplane (also called a linear separator), but more complex separators can be conceived, and these separators can be applied using kernel functions. The data points that define the hypersurface are called support vectors.

[0052] Support vector classifiers select a separator, where the separator is as far away as possible from the nearest data points, thus separating feature vector points associated with objects in a given class from those associated with objects outside that class. For support vector regression, a high-dimensional tube with an acceptable error radius is constructed that maximizes the flatness of the associated curve or function while minimizing the error of the dataset. In other words, the tube is the envelope around the fitted curve, defined by the set of data points closest to the curve or surface.

[0053] The advantage of Support Vector Machines (SVMs) is that once support vectors have been identified, the remaining observations can be removed from the computation, significantly reducing the computational complexity. SVMs typically operate in two phases: a training phase and a testing phase. During the training phase, a set of support vectors is generated to execute decision rules. During the testing phase, decisions are made using the decision rules. Support vector algorithms are methods used to train SVMs. By executing this algorithm, a training set of parameters is generated, including the support vectors characterizing the SVM. Representative examples of support vector algorithms suitable for this embodiment include, but are not limited to, sequence minimization optimization.

[0054] In KNN analysis, the affinity or proximity of objects is determined. Affinity is also known as the distance between objects in the feature space. Based on the determined distances, objects are clustered and outliers are detected. Therefore, KNN analysis is a technique for finding distance-based outliers based on the distance of an object to its k-th nearest neighbor in the feature space. Specifically, each object is ranked based on its distance to its k-th nearest neighbor. The farthest object is declared an outlier. In some cases, multiple farthest objects are declared outliers. That is, for parameters such as k neighbors and a specified distance, if the number of objects at or less than a specified distance from an object does not exceed k, then that object is an outlier. KNN analysis is a classification technique that uses supervised learning. A category is presented and compared to a training set with two or more categories. The category is assigned to the category most common among its k nearest neighbors. That is, the distances to all categories in the training set are calculated to find the k nearest neighbors, and the majority category is extracted from the k nearest neighbors and assigned to that category.

[0055] Association rule algorithms are techniques used to extract meaningful association patterns between features.

[0056] In the context of machine learning, the term "association" refers to any relationship between features, not just relationships that predict a particular classification or value. Associations include, but are not limited to, finding association rules, finding patterns, performing feature evaluation, performing feature subset selection, developing predictive models, and understanding the interactions between features.

[0057] The term "association rule" refers to elements that frequently co-occur in a dataset. It includes, but is not limited to, association patterns, discriminative patterns, frequent patterns, closed patterns, and mega-patterns.

[0058] The main steps in a common association rule algorithm are to find the most frequent set of categories or features among all observations. Once the list is obtained, rules can be extracted from it.

[0059] The self-organizing map described above is an unsupervised learning technique commonly used for the visualization and analysis of high-dimensional data. A typical application focuses on visualizing central dependencies within data in a plot. The plot generated by the algorithm can be used to accelerate the identification of association rules by other algorithms. This algorithm typically consists of a grid of processing units called "neurons." Each neuron is associated with a feature vector called an observation. The plot attempts to represent all available observations with optimal accuracy using a restricted set of models. Simultaneously, the models are ordered on the grid such that similar models are close to each other, and dissimilar models are far apart. This procedure enables the identification and visualization of dependencies or associations between features in the data.

[0060] Feature evaluation algorithms aim to rank features or select features based on their impact after ranking.

[0061] Information gain is one of the machine learning methods applicable to feature evaluation. The definition of information gain requires the definition of entropy, which is a measure of impurities in a training instance set. The amount by which the entropy of a target feature decreases by knowing the value of a certain feature is called information gain. Information gain can be used as a parameter to determine the effectiveness of a feature in explaining a treatment response. According to some embodiments of the invention, symmetric uncertainty is an algorithm that can be used by feature selection algorithms. Symmetric uncertainty compensates for the bias of information gain towards features with more values ​​by normalizing features to the [0,1] range.

[0062] Subset selection algorithms rely on a combination of evaluation and search algorithms. Similar to feature evaluation algorithms, subset selection algorithms rank subsets of features. However, unlike feature evaluation algorithms, the subset selection algorithm used in this embodiment aims to select the feature subset that has the highest impact on subject responses while considering the redundancy among the features contained in the feature subset. The benefits of feature subset selection include improved data visualization and understanding, reduced measurement and storage requirements, reduced training and usage time, and the elimination of distracting features to improve classification.

[0063] The two basic approaches to subset selection algorithms are the processes of adding features to a working subset (forward selection) and removing features from the current feature subset (backward elimination). In machine learning, forward selection is performed differently than in statistical procedures with the same name. In machine learning, the features to be added to the current subset are found by evaluating the performance of the current subset after adding a new feature using cross-validation. In forward selection, subsets are constructed by sequentially adding each remaining feature to the current subset, while cross-validation is used to evaluate the expected performance of each new subset. Features that lead to the best performance when added to the current subset are retained, and the process continues. The search ends when none of the remaining available features can improve the predictive power of the current subset. This process finds a locally optimal feature set.

[0064] Backward elimination is implemented in a similar manner. In the case of backward elimination, the search ends when further reduction of the feature set fails to improve the predictive power of the subset. This embodiment envisions search algorithms for forward, backward, or bidirectional searches. Representative examples of search algorithms suitable for this embodiment include, but are not limited to, exhaustive search, greedy hill climbing, random perturbation of subsets, wrapper algorithms, probabilistic speed search, pattern search, ranking speed search, and Bayesian classifiers.

[0065] Decision trees are decision support algorithms that form a logical path of steps involved in considering inputs to make a decision.

[0066] The term "decision tree" refers to any type of tree-based learning algorithm, including but not limited to model trees, classification trees, and regression trees.

[0067] Decision trees can be used to hierarchically classify datasets or their relationships. A decision tree has a tree structure including branch nodes and leaf nodes. Each branch node specifies an attribute (split attribute) and a test to be performed on the value of the split attribute (split test), branching to other nodes based on all possible outcomes of the split test. The branch node that serves as the root of the decision tree is called the root node. Each leaf node can represent a category or a value. Leaf nodes can also contain additional information about the represented category, such as a confidence score, which measures the level of confidence in the represented category (i.e., the probability that the classification is accurate). For example, the confidence score can be a continuous value ranging from 0 to 1, where a score of 0 indicates very low confidence (e.g., the represented category has a very low indication value), and a score of 1 indicates very high confidence (e.g., the represented category is almost certainly accurate).

[0068] According to some embodiments of the present invention, regression techniques that can be used include, but are not limited to, linear regression, multiple regression, logistic regression, probabilistic unit regression, ordered logistic regression, ordered probabilistic unit regression, Poisson regression, negative binomial regression, multinomial logistic regression (MLR), and truncated regression.

[0069] Logistic regression, or log-odds regression, is a type of regression analysis used to predict outcomes for a categorical dependent variable (a dependent variable that can take a finite number of values, where the magnitude of the values ​​is not significant, but the order of their magnitudes may or may not be significant) based on one or more predictor variables. Logistic regression can also predict the probability of occurrence for each data point. There are also multinomial variants of logistic regression. Multinomial logistic regression models are regression models that generalize logistic regression by allowing more than two discrete outcomes. That is, it is a model used to predict the probabilities of different possible outcomes for a categorically distributed dependent variable given a set of independent variables (which can be real values, binary values, categorical values, etc.). For binary variables, the Youden index is typically used to determine the critical value between 0 and 1 associations.

[0070] A Bayesian network is a model representing variables and the conditional dependencies between them. In a Bayesian network, variables are represented as nodes, and nodes can be connected to each other through one or more links. A link indicates the relationship between two nodes. Nodes typically have corresponding conditional probability tables, which are used to determine the probability of a node's state given the states of the other nodes it is connected to. In some embodiments, a Bayesian optimal classifier algorithm is employed to apply the maximum a posteriori hypothesis to new records to predict their classification probability, and to calculate probabilities based on each of the other hypotheses obtained from the training set, using these probabilities as weighting factors for future predictions of subject responses. Algorithms suitable for searching for the optimal Bayesian network include, but are not limited to, algorithms based on global score metrics. An alternative approach to building the network is to use a Markov blanket. A Markov blanket isolates nodes from the influence of any nodes outside a boundary consisting of the node's parent, child, and the parent of each child node.

[0071] Instance-based techniques generate new models for each instance, rather than making predictions based on trees or networks generated from the training set (once).

[0072] In the context of machine learning, the term "instance" refers to an example from a dataset.

[0073] Instance-based techniques typically store the entire dataset in memory and build a model from a set of records similar to the record being tested. This similarity can be evaluated, for example, through nearest neighbor or locally weighted methods, such as using Euclidean distance. Once a set of records is selected, several different techniques (such as Naive Bayes) can be used to build the final model.

[0074] Neural networks are a class of algorithms based on the concept of interconnected "neurons." In a typical neural network, neurons contain data values, where each data value influences the value of the connected neuron based on the strength of connections with a predefined strength and whether the sum of connections for each particular neuron reaches a predefined threshold. By determining appropriate connection strengths and thresholds (a process also known as training), neural networks can achieve efficient recognition of images and characters. Typically, these neurons are grouped into layers to make the connections between groups more explicit and facilitate the computation of each value. Each layer of the network can have a different number of neurons, and these neurons may or may not be related to a specific quality of the input data.

[0075] In one implementation known as a fully connected neural network, each neuron in a particular layer connects to neurons in the next layer and provides input values ​​to those neurons. These input values ​​are then summed, and the sum is compared to a bias term or threshold. If the value exceeds the threshold for a particular neuron, that neuron has a positive value, which can be used as input to neurons in the next layer. This computation continues across the layers of the neural network until the final layer is reached. At this point, the output of the neural network routine can be read from the values ​​in the final layer. Unlike fully connected neural networks, convolutional neural networks operate by associating arrays of values ​​with each neuron rather than individual values. The transformation of neuron values ​​in subsequent layers generalizes from multiplication to convolution.

[0076] The machine learning programs used in some embodiments of the invention are trained machine learning programs. According to some embodiments of the invention, the machine learning program can be trained by feeding a microbiome data of a subject population, for whom responses to food categories have been determined by blood tests. Once the data is fed, the machine learning training program generates a trained machine learning program of the selected type, which can then be used without retraining.

[0077] The machine learning procedures applicable to this embodiment are described in International Publication No. WO2015 / 166489 and in the following reference: [Zeevi et al. Personalized nutrition by prediction of glycemic responses. Cell. 2015; Vol. 163 (No. 5): pp. 1079-1094. doi:10.1016 / j.cell.2015.11.001], the contents of which are hereby incorporated by reference.

[0078] Therefore, the trained machine learning program receives a set of subject descriptor features and identified food categories as input, and provides a score as output indicating the subject's predicted response to the identified food categories. When multiple different identified food categories exist, the trained machine learning program is preferably applied individually to each of the food categories to provide a separate score for each food category.

[0079] In some embodiments of the invention, the method proceeds to step 15, wherein a subject's specific response to a diet comprising two or more food categories is predicted. This can be accomplished by predicting a response to each food category constituting the diet to provide a score for each individual food category, and then combining the scores of the individual food categories to provide a diet score. The scores of the individual food categories can be combined by averaging or using any other statistical measure of the combined scores. Preferably, the method employs weights to perform this combination. Typically, the weights are based on the relative quantities (e.g., mass, volume) of the individual food categories constituting the diet. For example, suppose the diet consists of m1 grams of food category f1, m2 grams of food category f2, and m3 grams of food category f3, and the scores for these food categories are s1, s2, and s3, respectively. Further suppose the diet score is calculated as a weighted average of s1, s2, and s3. In this case, the diet score is (m1·s1 + m2·s2 + m3·s3) / (m1 + m2 + m3).

[0080] The food categories in the diet can be those identified at 13 locations. Alternatively, the diet can consist of one or more food categories identified at 13 locations and one or more additional food categories provided by the subject to the server. Typically, the additional food categories are selected by the subject at a remote location from a predefined list of food categories. For example, the subject can operate a graphical user interface (GUI) to select food categories. The GUI can be the GUI of a webpage or a mobile app. Preferably, the GUI includes a display of multiple food categories and corresponding multiple GUI selection controls that allow the subject to select individual food categories one by one or in groups to define the diet. The controls activated by the subject can be transmitted from the GUI to a data processor of a computer or mobile device, and the data processor can process the controls, identify which food categories the subject has selected, and transmit the data related to these food categories to the server via a communication network.

[0081] Once one or more scores are obtained, the method can proceed to step 16, where at least a portion of the received image data is superimposed on one or more scores obtained at step 14 and optionally also at step 15, thereby generating processed image data. When scores exist for each of the multiple food categories, the received image data, or a portion thereof, is optionally and preferably superimposed on each of these scores. Embodiments are also envisioned in which the received image data, or a portion thereof, is superimposed on each score for each food category, but not on a dietary score that can be transmitted to the subject through a different channel.

[0082] In some embodiments of the invention, the method proceeds to step 17, wherein the received image data, or a portion thereof, is further overlaid with food category identification information. This identification information may be text identification information or recognizable symbols.

[0083] The overlay at 16 and 17 is preferably done in such a way that once the processed image data is displayed, the score and optional identification information are visually associated with the image of the corresponding food category (e.g., displayed near the image of the food category or with a pointer to the image of the food category).

[0084] The method can then proceed to 18, wherein the processed image data is transmitted to a display device that can be viewed by a subject at a remote location. For example, a mobile app may have a live viewing mode in which the app displays an image captured by the mobile device's camera on the mobile device's display, but this image is overlaid with scores and optionally recognition information as further detailed above.

[0085] The subject can then operate the app to switch between screens displaying a list of food categories that constitute the meal, and update the meal using one or more of the food categories identified on the processed image. The server can then obtain the updated meal from the app, calculate the meal score as further detailed above, and transmit it back to the app for display on the monitor. Alternatively or additionally, the subject can select the list of food categories before capturing the image, in which case the server can transmit both the scores of the individual food categories and the meal score to the app.

[0086] The method ends at point 19.

[0087] The method of this embodiment can be executed via a server-client configuration according to some embodiments of the present invention, as will now be referred to. Figure 2 Explanation.

[0088] Figure 2 The illustration depicts a client computer 30 with a hardware processor 32, which typically includes input / output (I / O) circuitry 34, a hardware central processing unit (CPU) 36 (e.g., a hardware microprocessor), and hardware memory 38, which typically includes both volatile and non-volatile memory. The CPU 36 communicates with the I / O circuitry 34 and the memory 38. The client computer 30 preferably includes a user interface, such as a graphical user interface (GUI) 42, that communicates with the processor 32. The I / O circuitry 34 preferably communicates with the GUI 42 in a suitably structured manner. The client computer 30 also communicates with a camera 46, which can be used to capture scene images featuring one or more food items or their representations.

[0089] A server computer 50 is also shown, which may similarly include a hardware processor 52, I / O circuitry 54, a hardware CPU 56, and hardware memory 58. The I / O circuitry 34 and 54 of the client computer 30 and server computer 50 preferably operate as transceivers that communicate with each other via wired or wireless communication. For example, the client computer 30 and server computer 50 may communicate via a network 40, such as a local area network (LAN), a wide area network (WAN), or the Internet. In some embodiments, the server computer 50 may be part of a cloud computing resource of a cloud computing facility that communicates with the client computer 30 via the network 40.

[0090] The GUI 42, processor 32, and optionally, and preferably, camera 46 can be integrated together in the same housing. Alternatively, they can be separate units that communicate with each other. The GUI 42 can optionally, and preferably, be part of a system including a dedicated CPU and I / O circuitry (not shown) to allow the GUI 42 to communicate with the processor 32. The processor 32 sends graphical and textual output generated by the CPU 36 to the GUI 42. The processor 32 also receives signals from the GUI 42 relating to control commands generated by the GUI 42 in response to user input. The processor 32 can also send an activation signal to the camera 46, which in turn can provide the processor 32 with signals carrying image data describing the images captured by the camera 46.

[0091] GUI42 can be of any type known in the art, such as, but not limited to, a keyboard and display, a touchscreen, etc. In a preferred embodiment, GUI42 is a GUI of a mobile device such as a smartphone, tablet computer, or smartwatch, and camera 46 is a camera of the mobile device. When GUI42 is a GUI of a mobile device, the CPU circuitry of the mobile device can act as processor 32 and can optionally and preferably execute the method by executing code instructions.

[0092] Client computer 30 and server computer 50 may further include one or more computer-readable storage media 44, 64, respectively. Media 44 and 64 are preferably non-transitory storage media storing computer code instructions for performing the methods of this embodiment, and processors 32 and 52 execute these code instructions. These code instructions can be executed by loading the corresponding code instructions into the respective execution memories 38 and 58 of the respective processors 32 and 52. Storage medium 64 preferably also stores one or more databases including psychologically annotated olfactory signatures, as further detailed above.

[0093] In operation, the subject operates camera 46 to capture images of a scene including one or more physical food categories or representations of one or more food categories. Typically, the subject launches an app on a mobile device that provides a set of activation controls. The subject operates the activation controls, which, for example, activate camera 46 in live view mode. The processor 32 of client computer 30 receives image data from camera 46 and transmits it to server computer 50 via network 40. The processor 52 of server computer 50 may identify the subject, for example, based on an authentication protocol between computers 30 and 50, and obtain, for example, a subject-specific set of subject descriptor features from medium 64. The processor 52 may execute object recognition instructions stored on medium 64 to analyze the image data and identify content related to food categories therein, and then feed the identified food category(s) and subject descriptor feature set to a trained machine learning program stored on medium 64 to provide a score describing the subject's response to the food categories, as further detailed above.

[0094] Then, processor 52 can generate processed image data, which includes a score superimposed on at least a portion of the received image data, and transmit the processed image data to client computer 30 to display the image data as an image on GUI 42. A representative example of a screen displayed by GUI 42 is shown in... Figures 3A to 3D The image shows various types of food categories 22, their corresponding scores 24, and the identified text 26. Figure 3D The display of dietary scores is also shown. In an exemplary embodiment, the dietary scores are overlaid on an image captured by the user. Figure 3E An embodiment is illustrated in which dietary scores are displayed on the screen of GUI 42, allowing subjects to select food categories from list 70, and also includes control 72 for allowing subjects to include (multiple) food categories identified from captured images into the list. In this case, dietary scores are provided as information control 28.

[0095] As used in this article, the term "about" means ±10%. The terms “comprises / comprising / includes / including”, “having”, and their cognates mean “including but not limited to”.

[0096] The term “composed of” means “including and limited to”.

[0097] The term "consistently of" means that a composition, method, or structure may include additional ingredients, steps, and / or portions, provided that such additional ingredients, steps, and / or portions do not substantially alter the fundamental and novel characteristics of the claimed composition, method, or structure.

[0098] As used herein, the singular forms “a / an” and “the” include plural indicators unless the context clearly indicates otherwise. For example, the terms “compound” or “at least one compound” can include multiple compounds, including mixtures thereof.

[0099] Throughout this application, various embodiments of the invention may be presented in a range format. It should be understood that the use of a range format is for convenience and brevity only and should not be construed as a rigid limitation on the scope of the invention. Therefore, a description of a range should be considered as specifically disclosing all possible subranges and the individual numerical values ​​within that range. For example, a description of a range from 1 to 6 should be considered as specifically disclosing subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., and the individual numbers within that range, such as 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.

[0100] Whenever a range of values ​​is indicated in this document, it is intended to include any referenced numerical value (fraction or integer) within the indicated range. The phrases “range between the first indicated number and the second indicated number” and “range from the first indicated number to the second indicated number” are used interchangeably in this document and are intended to include the first indicated number and the second indicated number, as well as all fractional and integer values ​​in between.

[0101] It should be understood that, for clarity, certain features of the invention described in the context of a single embodiment may also be provided in combination in a single embodiment. Conversely, for brevity, various features of the invention described in the context of a single embodiment may also be provided individually or in any suitable sub-combination, or in the suitable form as described in any other described embodiment of the invention. Certain features described in the context of various embodiments should not be considered essential features of those embodiments unless the embodiment would not function without those elements.

[0102] Although the invention has been described in conjunction with specific embodiments thereof, it will be apparent to those skilled in the art that many alternatives, modifications, and variations will be readily apparent. Therefore, it is intended to cover all such alternatives, modifications, and variations falling within the spirit and broad scope of the appended claims.

[0103] The intention of the applicants is that all publications, patents, and patent applications mentioned in this specification are incorporated herein by reference in their entirety, as if each individual publication, patent, or patent application had been expressly and individually indicated as being incorporated herein by reference. Furthermore, any reference or identification of any reference in this application should not be construed as an admission that such reference is prior art to the invention. The use of section headings should not be construed as necessarily limiting. Additionally, any priority documents(s) of this application are hereby incorporated herein by reference in their entirety.

Claims

1. A method of processing an image, comprising: receiving image data from a subject at a remote location; analysing the image data to identify content therein relating to food items; obtaining a set of subject descriptor features specific to the subject; applying a trained machine learning procedure to the food items and the set of features to provide a score describing the subject's reaction to the food items; generating processed image data comprising the score superimposed on at least part of the received image data; and transmitting the image data to a display device viewable by the subject at the remote location. The analysing the image data comprises identifying non-textual content in the image.

2. The method of claim 1, wherein, The analysing the image data comprises applying character recognition to textual content in the image.

3. The method of any one of claims 1-2, wherein, The image data comprises a stream of image data and the processed image data comprises a stream of processed image data.

4. The method of any one of claims 1 to 3, wherein, The generating the processed image data further comprises superimposing identifying information of the food items.

5. The method of any one of claims 1 to 4, wherein, The analysing the image data comprises identifying different content relating to different food items in the image data, wherein the applying the trained machine learning procedure comprises applying the trained machine learning procedure separately to each of the food items to provide a score describing the subject's reaction to each of the food items, and wherein the generating the processed image data comprises superimposing each of the scores on at least part of the received image.

6. The method of any one of claims 1 to 5, wherein, At least two of the different content are in a single image frame.

7. The method of claim 6, wherein, At least two of the different content are in different image frames.

8. The method of claim 6, wherein, 9. The method of any of claims 6 to 8, comprising calculating a meal score for a selection of the food items and transmitting the meal score to the display device. The selection comprises all of the food items.

10. The method of claim 9, wherein, The selection comprises a portion of the food items.

11. The method of claim 9, wherein, The portion is automatically selected.

12. The method of claim 11, wherein, 13. The method of claim 11, comprising receiving the portion from the subject at the remote location.

14. A computer software product comprising a computer readable medium having stored thereon program instructions that, when read by a data processor, cause the data processor to perform the method of any of claims 1 to 13.

15. A server system for processing an image, the server system comprising: a transceiver arranged to receive and transmit image data over a communications network; and a processor arranged to communicate with the transceiver and perform the method of any of claims 1 to 14.

16. A system for processing an image, comprising: an imaging device for capturing image data from a scene; a display device; and a server system as claimed in claim 15. ​ ​ a data processor configured to transmit the image data to a server, receive processed image data from the server, wherein the captured image data is overlaid with scores describing a subject's response to a food category or representation of a food category contained in the captured image data, and display a graphical user interface (GUI) containing an image corresponding to the processed image data on the display device.

17. The system of claim 16, wherein, the data processor is configured to receive a selection of the food category from the GUI, add the selection to a list of food categories defining a meal, transmit the list to the server, responsively receive a meal score for the list from the server, and display the meal score on the GUI.

18. A method of displaying processed image content, comprising: capturing image data from a scene containing a food category or representation of a food category; transmitting the image data to a server; receiving processed image data from the server, wherein the captured image data is overlaid with scores describing a subject's response to the food category; and displaying a graphical user interface (GUI) containing an image corresponding to the processed image data on a display device.

19. The method of claim 18, wherein, the capturing the image data includes capturing an image of a food category.

20. The method of claim 18, wherein, the capturing the image data includes capturing an image of a food menu containing textual content describing a plurality of food categories.

Images