Mapping machine learning models to answer queries according to semantic specifications

Abstract

Automatically mapping and applying machine learning models in accordance with semantic specifications to answer queries. A query is parsed to extract keywords from the query and contextualize the query. Based on the keywords, machine learning models that process concepts associated with the keywords are selected. The machine learning models are ordered in accordance with the contextualization of the query. The machine learning models are run on multi-modal data in an order in accordance with the ordering, wherein data resulting from the output of one of the machine learning models is used as input to another of the machine learning models. Based on the results of running the machine learning models, query results are output.

Description

TECHNICAL FIELD

[0001] The present application relates generally to computer and computer applications, machine learning, automated question answering, search engines, and more particularly to application of automatically mapping and / or combining machine learning models according to semantic specifications to answer queries. BACKGROUND

[0002] Currently, there is a lack of mechanisms to extract semantic information from multi-modal content using symbolic representations and using a combination of multiple machine learning models to answer user queries. Specifically, current solutions attempt to index all content and store both types of information in a database. In this case, indexing is done manually through human annotation or statically through models that can be applied to extract concepts regardless of where the concepts are extracted from. SUMMARY

[0003] The summary of the disclosure is given to help understand the computer system and method, for example, of automatically mapping and / or combining machine learning models according to semantic specifications to answer queries, rather than to limit the disclosure or the invention. It should be understood that various aspects and features of the disclosure can be advantageously used alone or in combination with other aspects and features of the disclosure in some cases, or in other cases. Therefore, changes and modifications can be made to the computer system and / or the method of its operation to achieve different effects.

[0004] In one aspect, a system can include a processor and a storage device coupled with the processor. The processor can be configured to receive a query. The processor can also be configured to parse the query, extract keywords from the query, and contextualize the query. The processor can also be configured to select, based on the keywords, machine learning models that process concepts associated with the keywords. The processor can also be configured to rank the machine learning models according to the contextualization of the query. The processor can also be configured to run the machine learning models on multi-modal data in an order according to the ranking, wherein data resulting from the output of one of the machine learning models is used as input to another of the machine learning models. The processor can also be configured to output a result of the query based on results of running the machine learning models.

[0005] In another aspect, a computer-implemented method can include receiving a query. The method can also include parsing the query, extracting keywords from the query, and contextualizing the query. The method can also include selecting machine learning models that process concepts associated with the keywords based on the keywords. The method can also include ranking the machine learning models according to the contextualization of the query. The method can also include running the machine learning models in an order according to the ranking on multi-modal unstructured data, where data resulting from an output of one of the machine learning models is used as an input to another of the machine learning models. The method can also include outputting a result of the query based on results of running the machine learning models in the order according to the ranking.

[0006] A computer readable storage medium storing a program of instructions executable by a machine to perform one or more of the methods described herein can also be provided.

[0007] Further features and features of the various embodiments, and their structure and operation, are described in detail below with reference to the accompanying drawings. The same reference numbers in different drawings represent the same or functionally similar elements. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a diagram illustrating a system architecture in an embodiment.

[0009] Figure 2 is a flow diagram illustrating a method for model injection and knowledge construction in an embodiment.

[0010] Figure 3 is a flow diagram illustrating a query answering method in an embodiment.

[0011] Figure 4 is a diagram illustrating an example of a knowledge graph in an embodiment.

[0012] Figure 5 is another flow diagram illustrating a method in an embodiment.

[0013] Figure 6 is a diagram illustrating components of a system that can automatically map and combine machine learning models according to semantic specifications to answer queries in an embodiment.

[0014] Figure 7 shows a schematic diagram of an example computer or processing system that can implement a system in an embodiment.

[0015] Figure 8 shows a cloud computing environment in an embodiment.

[0016] Figure 9 shows a set of functional abstraction layers provided by a cloud computing environment in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] In one or more embodiments, artificial intelligence (AI) systems and methods can automatically map, contextualize, and combine applications of machine learning models to answer queries in accordance with semantic specifications. The systems and methods may, for example, contextualize and semantically orchestrate relationships between machine learning (ML) models and multi-modal data and symbolic concept labels when answering queries. For example, the system can describe neural-symbolic integration through a knowledge representation that specifies the entire ML workflow, including what each model consumes and produces. The system can infer these relationships through multi-modal data and appropriate models. With another functionality that maps this semantic information to a selection of machine learning models, the system can also process user-specified queries to extract symbolic concept content. For example, the system can select, rank, and apply a hierarchy of machine learning models on specific multi-modal data and concept symbolic representations to dynamically index multi-modal data snippets when answering queries.

[0018] In answering queries, the search or search engine in embodiments of the systems and / or methods disclosed herein can consider the hierarchy and contextualization when performing information extraction mechanisms. For example, the search or search engine can extract meaning or semantics from the query and consider the mapping between the ML model functionality and the semantic description of the query.

[0019] The following illustrates some example scenarios. In a first example scenario, consider a user query that displays an image of an "XYZ mug," which means that the user is searching for images of a mug that displays the "XYZ" logo on the mug. In an embodiment, the system parses the query and determines that the user simply wants images. The system also knows that the user wants to include two concepts in the images. The system then, in a contextualized manner, finds a model that can extract the XYZ logo or word from the image and a model that can recognize a mug in the image. That is, there is a hierarchy in the model selection and orchestration. For example, the system first executes a mug identifier on all previously unprocessed images. Then, using the mug bounding boxes in the images, the system passes these cropped images to the next model, which is an XYZ logo or word identifier. The system then displays the results through a dashboard. In this way, the search results can be more focused and accurate. For example, in the example above, search results that include both the XYZ logo and a mug, but in a non-contextualized manner (e.g., a document includes the word or logo "XYZ," but as an instance separate from the image, e.g., there is no logo in the image of the mug) can be eliminated.

[0020] The same logic can be applied to different domains. For example, in a second example scenario, a geoscientist queries the system for seismic images containing a geological pattern called "mini-basin", but specifically, searching for mini-basins with converging strata. In this case, the system selects three models. The first model is to classify the geological type of the image to filter and only pass seismic images. The second model is a mini-basin classifier. Then, the system applies the third model to find the converging strata pattern using the bounding box of the mini-basin. Then, the system displays the results.

[0021] In a third possible example scenario, a user requests all impressionist landscape paintings by British painters. In a similar way, the system selects and orchestrates three models: a painter classifier, an art movement classifier, and a genre classifier.

[0022] The systems and / or methods disclosed herein can support ML model injection and association with their semantic descriptions, and can map semantic descriptions from queries to a hierarchically and contextualized set of available and applicable ML models. The systems and / or methods can execute the selected models at query time. In embodiments, the systems and / or methods can automatically map and combine applications of machine learning models according to semantic specifications to answer queries. The systems and / or methods can describe the relevant semantics and various possibilities of machine learning models in a knowledge representation, process queries to extract keywords and meanings, and map to contextualized machine learning models; select, rank, and apply selected machine learning models on selected multi-modal data and conceptual symbols; index multi-modal data snippets and machine learning models dynamically at query time.

[0023] Figure 1 FIG. 1 is a diagram illustrating a system architecture in one embodiment, which can implement a knowledge-oriented ML based question answering (QA) system. The illustrated components include computer-implemented components, e.g., components implemented and / or run on one or more hardware processors, or components coupled with one or more hardware processors. For example, the one or more hardware processors can include components such as programmable logic devices, microcontrollers, memory devices, and / or other hardware components, which can be configured to perform respective tasks described in this disclosure. The coupled memory devices can be configured to selectively store instructions executable by the one or more hardware processors.

[0024] The processor can be a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), another suitable processing component or device, or a combination thereof. The processor can be coupled with a memory device. The memory device can include random access memory (RAM), read-only memory (ROM), or another memory device, and can store data and / or processor instructions for implementing various functions associated with the methods and / or systems described herein. The processor can execute computer instructions stored in the memory or received from another computer device or medium.

[0025] The dashboard graphical user interface (GUI) 102 can be a user interface module or program that supports interaction between a user and an automated processor or computer. For example, the dashboard GUI 102 can be a browser or a browser-enabled interface, or another computer application. A user can input a search or query through the dashboard GUI 102, and the system can output the results of the search through the dashboard GUI 102.

[0026] The query parser 104 can be a natural language processing module or program that parses a query (e.g., a user-entered or user-specified search query). The query parser 104 can tokenize the search query into tokens for processing. The concept mapper 106 can be a natural language processing module or another program that can analyze the parsed query and determine one or more concepts specified in the query.

[0027] The model selector 112 can be a module or program that searches a database of machine learning models, such as the non-symbolic repository 116, and selects machine learning models that are relevant to or process the concepts determined by the concept mapper 106. The non-symbolic repository 116 can also store any other non-symbolic or unstructured data, such as documents, images, videos, and / or others.

[0028] The query engine 118 can be a computer module or program that performs a search by applying a combination of relevant machine learning models determined by the query structure. For example, a search can be performed by running one or more selected machine learning models on unstructured data (e.g., text, images, videos, or other textual documents, objects (e.g., images, videos, and / or multi-media documents)). Such unstructured data can also be stored in the non-symbolic repository 116. The machine learning models can be run in a contextualized order to accommodate the query. In embodiments, the query engine is responsible for testing stored data against selected machine learning models relevant to the query. The input to the query engine 118 can be a list of associated ML models, and the concepts identified in the query. The query engine 118 can output data deemed relevant after being classified by the ML models, which can be performed in the model orchestrator 120.

[0029] The model inspector 122 can be a computer module or program that can extract information from ML models, such as the parameters of the model, the input / output and other signatures. This information can be extracted from the documentation associated with the model.

[0030] The model orchestrator 120 can be a computer module or program that can combine ML models, for example, to run selected ML models in a particular order, for example, the output of one model can be used as input to another model in order to obtain data displayed as a search result. The model orchestrator 120 can analyze each selected model and determine the input and output each model takes and produces. Based on this analysis, the model orchestrator 120 can organize or determine in what order which model is run, and for example, which model’s output should be used for which model’s input.

[0031] The knowledge graph 110 can be a data structure comprising nodes and edges that stores concepts and relationships between concepts. Nodes can represent concepts, and edges connecting nodes can represent relationships between the nodes that the edges connect. In embodiments, edges can be weighted to represent the strength of the relationship between the edges. In embodiments, an existing knowledge graph can be used and augmented with additional information, additional nodes and edges. In another embodiment, for example, an existing dictionary or ontology that provides relationships between concepts can be used to construct the knowledge graph. In short, an ontology provides a description of relationships between concepts (e.g., entities). An example snippet of a knowledge graph is shown in Figure 4

[0032] Referring to Figure 1 ​The knowledge constructor 108 can be a computer module or program that builds the knowledge graph 110, or augments the knowledge graph 110 with information. ML models identified from the non-symbolic repository 116 can be linked to concepts or nodes in the knowledge graph 110. In embodiments, the knowledge constructor 108 can be responsible for creating symbolic descriptions of the ML models, associating these descriptions with concepts and relations of the ontology in the knowledge graph. For example, the knowledge constructor 108 can create a structure for the model.

[0033] The graph processor 114 can perform functions on graphs such as the knowledge graph 110. For example, information can be linked to nodes in the knowledge graph, such as ML models, which can process the concepts represented by the nodes. The graph processor 114 can take data incorporated in the answer set, structure the data and provenance aspects (combination of ML models used) in the knowledge graph, thereby creating corresponding symbolic descriptions.

[0034] Referring to the above example scenario, a user can input a search query for an image of a mug with an XYZ logo through the dashboard GUI 102. The query parser 104 parses the query. The concept mapper 106 determines concepts, such as "mug" and "XYZ" logo, and the model selector 112 selects a set of machine learning models from the non-symbolic repository 116 that can perform the query. The model orchestrator 120 contextualizes and combines execution of the machine learning models. For example, a model trained to recognize mug images can be run to find an image or bounding box of a mug image. Then, the model orchestrator 120 can pass only the cropped images with the mug to the next model, such as an XYZ logo identifier. The output of this model can be used as a search result, for example, presented or displayed through the dashboard GUI 102.

[0035] Similarly, referring to the above second example scenario, a user can input a query through the dashboard 102 to search for seismic images that exist in a small basin with convergent strata. The query parser 104 parses the query, the concept mapper 106 extracts concepts from the query, and the model selector 112 finds a set of models from the non-symbolic repository 116 that can handle the query. The model orchestrator 120 contextualizes and combines execution of the models. For example, the system can use a seismic image type classifier to find seismic images. Then, the seismic images can be passed to a model that can find small basin geological structures. Then, the system can pass only the cropped images with the small basin to the next model, such as a convergent strata classifier. The results can be presented or displayed through the dashboard GUI 102.

[0036] Figure 1The illustrated system can similarly provide search results for the example scenario of the query for impressionist landscape paintings of British painters. Different ML models, such as a painter classifier, an art movement classifier, and a genre classifier, can be selected and run in the order determined by the model orchestrator 120 to provide the results.

[0037] Figure 2 is a flowchart illustrating a method for model injection and knowledge construction in embodiments. The method can be run on or implemented by a computer processor, e.g., including a hardware processor. At 202, a model can be received. For example, a user can access a dashboard (e.g., 102 in Figure 1 ) to inject a model. The user can specify a model (e.g., a deep learning model, a neural network, another ML model) that is trained to perform a particular task (e.g., recognize a particular type of image, classify a particular object, text, and / or other). The user can also specify or input information associated with the model, e.g., the input the model can accept, the output the model can produce, parameters and other signatures, and other information associated with the model.

[0038] On the other hand, at 204, the computer processor can automatically extract such information about the injected model from, e.g., available documentation associated with the injected model. For example, a model inspector component or module (e.g., 122) can process the injected model to extract information, e.g., parameters, signatures, and / or other, from the model. Such information can be included in the documentation associated with the model that the model inspector can automatically extract. Figure 1

[0039] At 206, the computer processor can use the extracted information about the model to automatically create a symbolic description (structured description) of the model and associate the description with concepts and relationships of an ontology in a knowledge graph. For example, a knowledge graph can be established that includes concepts and relationships between the concepts. Such a knowledge graph can be represented in a data structure of nodes and connecting edges, where the nodes represent the concepts and the edges represent the relationships. The symbolic or structured description of the model can also be linked or connected to the concepts in the knowledge graph. For example, a knowledge constructor (e.g., 108) can create the symbolic description of the model with the extracted information and associate the description with one or more concepts in a knowledge graph (e.g., 110). Figure 1 Figure 1

[0040] At 208, the modified knowledge graph can be displayed. For example, a dashboard GUI (e.g., 102) can display the modified knowledge graph. Figure 1 ​​​The modified knowledge graph can be displayed or presented, thereby allowing ontology alignment, user curation, and learning through user interaction. For example, a user can provide feedback regarding the modified knowledge graph, further editing the knowledge graph.

[0041] At 210, it is determined whether another model is to be injected. If so, the logic continues at 202, otherwise, the logic can end, or return to its calling module.

[0042] Figure 3 is a flowchart illustrating a method of query answering in embodiments. The method can be run on or implemented by a computer processor, e.g., including a hardware processor. At 302, a query is received. For example, a user can access a dashboard GUI (e.g., Figure 1 , 102) to specify a query. At 304, the computer processor extracts keywords and the overall structure of the query, e.g., using natural language processing techniques. For example, a query parser (e.g., Figure 1 , 104) can extract keywords and the overall structure of the query.

[0043] At 306, the computer processor maps the extracted keywords into concepts and relationships in a knowledge graph, and uses the extracted structure to construct a corresponding query in knowledge graph terminology. For example, a concept mapper (e.g., Figure 1 , 106) maps the extracted keywords into concepts and relationships in a knowledge graph, and uses the extracted structure to construct a corresponding query in knowledge graph terminology.

[0044] At 308, the computer processor associates an ML model (e.g., from a repository) as a decision process for the particular concepts and relationships used in the query. For example, a model selector (e.g., Figure 1 , 112) associates an ML model (e.g., from a non-symbolic repository (e.g., Figure 1 , 116)) as a decision process for the particular concepts and relationships used in the query generated by the concept mapper (e.g., Figure 1 , 106).

[0045] At 310, the computer processor takes the query and the associated ML model, and evaluates the query through the knowledge graph. The evaluation process can include zero or more evaluation steps. For example, a query engine (e.g., Figure 1 , 118) takes the query and the associated ML model, and evaluates the query through the knowledge graph (e.g., Figure 1 , 110).

[0046] In embodiments, the query evaluation at 310 can include graph pattern-matching and connectivity tests. The query itself can be represented as a graph with "holes" and constraints. In embodiments, the task of the query engine is to find nodes and edges in the target graph that fill in these holes while satisfying the constraints. Specifically, the query engine can attempt to find all subgraphs of the target graph that are isomorphic (or in some cases homomorphic) to the query graph.

[0047] For example, to evaluate the query "select x where {x instanceOf Person}", the query engine first converts this to the graph "(x) -instanceOf-> (Person)", which has two nodes (x) and (Person), and one edge labeled "instanceOf" connecting the former to the latter. The query engine then searches the target graph for every node x for which there is a "instanceOf" edge from x to the node "Person". Any x that satisfies this test is a result of the query.

[0048] More complex queries can involve connectivity tests. For example, "select x where {x likes+ B}" means that not only does every x have to satisfy that there is a "likes" edge between x and "B", but—given the plus (+) operator in the query—every x has to satisfy that there is a non-zero length path of "subclass" edges connecting x and "B". The plus (+) operator introduces a connectivity test. The query in this example is asking for every x that likes B, or every x that likes someone who likes B, or every x that likes someone who likes someone who likes B, and so on.

[0049] In embodiments, a ML model can be used to decide whether a query restriction holds. For example, suppose there is a ML classifier that, given an image, can decide whether the image depicts a person. Suppose the query "select x where {x instanceOf Image, x depicts Person}" needs to be evaluated. In this case, the query engine will try to find every such x in the target graph, i.e., there is an edge "instanceOf" from x to "image" and an edge "depicts" from x to "person". Now, it is possible that there is an image I in the target graph that has not yet been analyzed, and all that is known about it is that it is an image. Thus, because the task here is to find every image with the given restriction ("depicts a person"), and because there is a model M in the ML library that, given an image, can decide whether the restriction holds, the query engine can use M to decide whether there should be an edge "depicts" from I to "person" when examining the unanalyzed image I. If, for a given I, the model M answers "yes", then in embodiments the query engine proceeds to add that edge to the graph (along with its provenance, i.e., the fact that the edge came from the evaluation of model M), and adds I to the result set.

[0050] At 312, it is determined whether there are more evaluation steps. For example, if there are other question components of the query that need to be answered, or other associated ML models that need to be run, then it can be determined that there are more evaluation steps. For example, the conjunctions in the composition of the query and the ML can be tested. As an example of a conjunction, consider the query "select x where {x instanceOf Image, x depicts Person, x depicts Cat}". That is, we want to find every image x that depicts a person and a cat. Suppose there are models Ml and M2 that can decide whether a person appears in an image and whether a cat appears in an image, respectively. When examining an unanalyzed image I, I must satisfy both restrictions to be included in the result set, and each of the restrictions can be tested by models Ml and M2. If Ml and M2 both answer "yes" when applied to I, i.e., if Ml(I) & M2(I) is true, then the query engine will include I in the result set. The "&" here is a conjunction operator.

[0051] As an example of composition, consider the query "select x where {x instanceOf Image, x depictsAnimal, x depicts Zebra}" and assume there are models M1 and M2 such that M1 can determine whether an image is an image of an animal (Animal) and M2 can determine whether an image of an animal is actually an image of a zebra (Zebra). If both models answer "yes", then the unanalyzed image I will satisfy this query, but the order in which the models are applied matters, because M2 can only be applied to animal images (unanalyzed image I is not a correct input for M2). In embodiments, the order of evaluation is determined by the input / output constraints of the models themselves as described in the knowledge graph. In this case, when examining image I, the query engine first applies M1, and only if M1 answers "yes" does it apply M2.

[0052] More specifically, in embodiments, the query engine can apply the models in a particular order determined from the signatures of the models: the method can represent M1 as a partial function from "images" to "animals" (from general images to animal images) and M2 as a partial function from "animals" to "zebras" (from animal images to zebra images). In symbolic notation, M1: Image -> Animal and M2: Animal -> Zebra. Given I of type "image", when examining the constraint "depicts a zebra", the query engine works backwards. The query engine knows that M2 can answer this question, but only if I is an animal image - which can be answered directly by M1. Thus, if M2(M1(I)) is defined, that is, if (M2*M1)(I) is defined, then the query engine will include I in the result set. By definition of function composition, the latter is not defined if M1(I) is not defined.

[0053] If there are more evaluation steps, at 314, the computer processor applies the combination of associated ML models (determined by the query structure) to the data in the repository and decides whether the examined data should be part of the result set. For example, the query engine (e.g., 118) uses the model orchestrator (e.g., 120) to apply the combination of associated ML models (determined by the query structure) to the data in the non-symbolic repository (e.g., 116) and decides whether the examined data should be part of the result set. Figure 1 Figure 1 Figure 1

[0054] ​​​At 316, the computer processor takes the data in the result set and adds it to the knowledge graph along with its provenance (the combination of ML models used to classify the data), creating a corresponding symbolic description. For example, the graphics processor (e.g., 114) takes the data in the result set and adds it to the knowledge graph (e.g., 110) along with its provenance (the combination of ML models used to classify it), creating a corresponding symbolic description. Figure 1 Figure 4 , 110), creating a corresponding symbolic description.

[0055] If at 312, there are no other evaluations to perform, then at 318, the computer processor presents the query results (the final result set), allowing further ontological alignment, user curation, and learning through user interaction. For example, the user can provide feedback regarding the query results. Based on the feedback, one or more ML models and the knowledge graph can be modified. For example, the query results can be presented or displayed through a dashboard GUI (e.g., 102). Figure 5

[0056] At 320, if there is another query to process, then the logic proceeds to 302. Otherwise, the logic can end or return to the calling process.

[0057] The inventive method performs query answering in one aspect through the dynamic, semantically guided combination of machine learning models that are incrementally (at query time) applied to data segments described in a contextualized knowledge graph.

[0058] Figure 6 is a diagram showing an example of a knowledge graph in an embodiment. The nodes of the knowledge graph 400 represent concepts, and the edges can represent relationships. By way of example, a particular painter node (e.g., including a proper name or identifier of a particular painter) 402, 404 can have a relationship to a particular country node 406, 408 (has citizenship). The particular painter node 402, 404 can also have a relationship to a profession node “painter” 414 (has profession). A particular painting node (e.g., can be referenced by a name or title or another identifier, and can include information such as style or type and other information) 410 can have a relationship to an object node “painting” 412 (e.g., subConceptOf). The particular painting can also have a relationship to a particular painter node (e.g., 402 or 404) (e.g., has creator). In an embodiment, the knowledge graph 400 can be updated based on a search. For example, through a search, it can be discovered that “painter A” 402 is the creator of “painting 1” 410. In this case, the knowledge graph 400 can be updated to include the relationship “has creator” between “painter A” 402 and “painting 1” 410. ​​

[0059] Figure 7 is another flowchart illustrating a method in an embodiment. The method can be run on or implemented by a computer processor, e.g., including a hardware processor. At 502, a query can be received. For example, a user, through a user interface or graphical user interface, can specify a question or query to a computer, e.g., to search for an answer or response to the query. At 504, the processor or computer processor parses the query and extracts keywords used in the query using natural language processing techniques. The processor also contextualizes the query by semantically analyzing the structure of the query. In an embodiment, the keywords are mapped to concepts in a knowledge graph, and the query can then be contextualized in terms of relationships of the concepts in the knowledge graph.

[0060] At 506, the processor searches, identifies, or selects machine learning models that can handle or process concepts associated with the keywords based on the keywords. For example, a machine learning model that handles a concept associated with a keyword in the query, another machine learning model that handles another concept associated with another keyword in the query, and so on, can be identified or selected. In an embodiment, the machine learning models are selected from a repository containing trained machine learning models. Examples of machine learning models can be neural networks, deep learning networks, and other unsupervised machine learning models, semi-supervised machine learning models, supervised machine learning models, and / or other handlers. In an embodiment, the machine learning models can be linked to or associated with respective concepts in the knowledge graph. In an embodiment, as an example, the machine learning models can include machine learning classifiers, each trained to classify a particular object. For example, each model can perform a different classification.

[0061] At 508, the processor or computer processor orders the machine learning models in accordance with the contextualized query or the contextualization. For example, if the query includes searching for an image of an item that displays a particular logo within the item, and a machine learning model that classifies images of items and another machine learning model that classifies images of logos are selected, the processor can determine to order the machine learning models in accordance with an order of the machine learning model that classifies images of items and the machine learning model that classifies images of logos.

[0062] At 510, the processor or computer processor runs the machine learning models on the multi-modal unstructured data in an ordering sequence, where data produced from the output of one of the machine learning models can be used as input to another of the machine learning models. Examples of multi-modal data can include, but are not limited to, textual data, audio data, video data, and image data. By way of example, a first machine learning model can take as input multi-modal unstructured data, and classify that data. An image classified by the first machine learning model as containing an item or object being searched for can be cropped and input to a second machine learning model to further classify the input to obtain a desired result. In embodiments, composition is used in any situation where the output of one model is used as input to another model, such as the case where one model crops a relevant portion of an image, which latter is then fed to a second or another model. For example, in the example involving animals and zebra images described above, when given an image of an animal, model Ml does not simply answer "yes" or "no", but can create a new image J by cropping the portion of the original image I where the animal appears and answer "yes" for J. The new image J will then be fed to M2, and if it is an image of a zebra, J will be included in the result set. This can be described backwards as: I can not be an image of a zebra, but can contain an image of a zebra. One way to find a suitable input for M2 is to examine sub-images of I using model Ml. At 512, based on running the machine learning models in the ordering sequence, a search result is output.

[0063] Figure 7is a diagram illustrating components of a system that can automatically map and combine machine learning models according to semantic specifications to answer queries in one embodiment. One or more hardware processors 602, such as central processing units (CPUs), graphics processing units (GPUs), and / or field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and / or another processor, can be coupled with a memory device 604 and perform query answering with machine learning models according to semantics of a query. The memory device 604 can include random access memory (RAM), read-only memory (ROM), or another memory device, and can store data and / or processor instructions for implementing various functions associated with the methods and / or systems described herein. The one or more processors 602 can execute computer instructions stored in the memory 604 or received from another computer device or medium. For example, the memory device 604 can store instructions and / or data for the one or more hardware processors 602 to work on, and can include an operating system and other instruction and / or data programs. The one or more hardware processors 602 can receive input including unstructured and / or multi-modal data, such as for classification by one or more machine learning models, for answering a query. In one aspect, the unstructured or non-symbolic multi-modal data can be stored in a storage device 606 or received from a remote device through a network interface 608, and can be temporarily loaded into the memory device 604 for classification. For example, one or more machine learning models can be stored on the memory device 604 for execution by the one or more hardware processors 602. The one or more hardware processors 602 can be coupled to interface devices, such as a network interface 608 for communicating with remote systems, for example, over a network, and an input / output interface 610 for communicating with input and / or output devices, such as a keyboard, a mouse, a display, and / or others.

[0064] Figure 8 A diagram illustrating a schematic of an example computer or processing system that can implement the system in one embodiment is shown. The computer system is only one example of a suitable processing system and is not intended to limit the scope of use or functionality of embodiments of the methods described herein. The processing system shown can be operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well- known computing systems, environments, and / or configurations that can be suitable for use with the processing system shown include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like. Figure 8 Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with the processing system shown include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like.

[0065] A computer system can be described in the general context of executable instructions (such as program modules) that run on it. Generally, a program module may include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. The computer system may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules may reside on local and remote computer system storage media, including memory storage devices.

[0066] The components of the computer system may include, but are not limited to, one or more processors or processing units 12, system memory 16, and a bus 14 that couples the various system components, including system memory 16, to processor 12. Processor 12 may include module 30 that performs the methods described herein. Module 30 may be programmed into an integrated circuit of processor 12 or loaded from memory 16, storage device 18, or network 24, or a combination thereof.

[0067] Bus 14 can represent any one or more of several types of bus architectures, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor or local buses using any of a variety of bus architectures. By way of example and not limitation, such architectures include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses.

[0068] Computer systems may include a variety of computer system-readable media. Such media can be any available media accessible to a computer system, and may include volatile and non-volatile media, removable and non-removable media.

[0069] System memory 16 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory or other memory. The computer system may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 18 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (e.g., a "hard disk drive"). Although not shown, a disk drive may be provided for reading from or writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from or writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media). In such a case, each may be connected to bus 14 via one or more data media interfaces.

[0070] The computer system can also communicate with one or more external devices 26 such as a keyboard, a pointing device, a display 28, etc.; one or more devices that enable a user to interact with the computer system; and / or any devices (e.g., network card, modem, etc.) that enable the computer system to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interfaces 20.

[0071] The computer system can also communicate with one or more networks 24 such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) through network adapter 22. As pictured, network adapter 22 communicates with the other components of the computer system through bus 14. It should be understood that, although not shown, other hardware and / or software components could be used in connection with the computer system. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0072] It is to be understood that while the present disclosure can include descriptions of cloud computing, the implementation of the teachings described herein are not limited to cloud computing environments. Rather, embodiments of the application are capable of being implemented in conjunction with any other type of computing environment now known or later developed. Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a provider of the service. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0073] The characteristics are as follows:

[0074] Broad network access: capabilities are available over the network and accessed through standard mechanisms that promote the use of heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0075] Resource pooling: the provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to consumer demand. There is a sense of location independence in that the consumer generally has no control or knowledge over the exact location of the provided resources but can be able to specify location at a higher level of abstraction (e.g., country, state, or data center).

[0076] Resource pooling: the provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to consumer demand. There is a sense of location independence in that the consumer generally has no control or knowledge over the exact location of the provided resources but can be able to specify location at a higher level of abstraction (e.g., country, state, or data center).

[0077] Fast elasticity: capability available in some cases to be rapidly provisioned and released in a few minutes with high initial cloud computing efficiency (e.g., minutes). The capability available for provisioning typically appears as unlimited and can be purchased in any quantity at any time.

[0078] Measured service: cloud systems automatically control and optimize resource use by leveraging usage by quantity of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported providing transparency for both the provider and consumer of the service.

[0079] Service models are as follows:

[0080] Software as a Service (SaaS): the capability provided to the consumer is to use the provider's applications running on a cloud infrastructure. The applications are accessible from various client devices through a thin client interface such as a web browser (e.g., web-based e-mail). The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0081] Platform as a Service (PaaS): the capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but has control over the deployed applications and possibly application hosting environment configurations.

[0082] Infrastructure as a Service (IaaS): the capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include an operating system and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, deployed applications, and possibly limited control of select networking components (e.g., host firewalls).

[0083] Deployment models are as follows:

[0084] Private cloud: the cloud infrastructure is operated solely for an organization. It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0085] Community cloud: the cloud infrastructure is shared by several organizations and supports mission-oriented business operations by a specific community of consumers. It can be managed by the organizations or a third party and can exist on-premises or off-premises.

[0086] Public cloud: Cloud infrastructure that is available to the public or large industry groups and is owned by organizations that sell cloud services.

[0087] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that maintain a unique entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported (e.g., cloud bursting for load balancing between clouds).

[0088] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure that includes a network of interconnected nodes.

[0089] See now Figure 9 The description illustrates a cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 to which local computing devices used by cloud consumers can communicate. These local computing devices include personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or in-vehicle computer systems 54N. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platforms, and / or software as services that cloud consumers do not need to maintain on their local computing devices. It should be understood that... Figure 8 The types of computing devices 54A-N shown are intended to be illustrative only. Computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0090] See now Figure 9 This demonstrates the 50 (cloud computing environment) ​ This provides a set of functional abstractions. It should be understood beforehand. ​ The components, layers, and functions shown are intended to be illustrative only, and embodiments of the invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0091] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a mainframe 61; a RISC (Reduced Instruction Set Computer) based server 62; a server 63; a blade server 64; a storage device 65; and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0092] Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71 ; virtual storage 72; virtual networks 73, including virtual private networks; virtual applications and operating systems 74; and virtual clients 75.

[0093] In one example, management layer 80 can provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources that are utilized to perform tasks within the cloud computing environment. Metering and Pricing 82 provide cost tracking as resources are utilized within the cloud computing environment, and billing or invoicing for consumption of these resources. In one example, these resources can include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management such that required service levels are met. Service Level Agreement (SLA) planning and fulfillment 85 provide pre-arrangement for, and procurement of, cloud computing resources for which a future requirement is anticipated in accordance with an SLA.

[0094] Workloads layer 90 provides examples of functionality for which the cloud computing environment can be utilized. Examples of workloads and functions which can be provided from this layer include: mapping and navigation 91 ; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics processing 94; transaction processing 95; and query processing 96.

[0095] The present application can be a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0096] A computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch cards or raised structures in grooves of a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0097] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0098] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for an integrated circuit, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and a procedural programming language such as the "C" programming language or the like. The computer readable program instructions can execute entirely on a user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0099] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0100] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0101] Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0102] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to the present application. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0103] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. It will be further understood that the terms "comprises" "comprising," "includes" "including," "has" "having," "contains" "containing," or "has" "having," when used herein, specify the presence of stated features, integers, steps, operations, elements, or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof. As used herein, the phrase "in an embodiment" does not necessarily refer to the same embodiment, although it may. As used herein, the phrase "in one embodiment" does not necessarily refer to the same embodiment, although it may. As used herein, the phrase "in another embodiment" does not necessarily refer to a different embodiment, although it may. Furthermore, embodiments and / or components of embodiments can be combined in any combination, unless the context clearly indicates otherwise.

[0104] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims that follow, if any, are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present application has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the application in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the application. The embodiment was chosen and described in order to best explain the principles of the application and the practical application, and to enable others skilled in the art to understand the application for various embodiments with various modifications as are suited to the particular use contemplated.

Claims

1. A computer-implemented method for performing query answering, comprising: receiving a query; parsing the query to extract keywords from the query and contextualizing the query; based on the keywords, selecting machine learning models capable of handling concepts associated with the keywords; ranking the machine learning models according to the contextualized query; running the machine learning models on multi-modal unstructured data in the ranked order, wherein data resulting from the output of one of the machine learning models is used as input to another of the machine learning models; outputting a result of the query according to the results of running the machine learning models in the ranked order.

2. The method of claim 1, wherein, mapping the keywords to concepts in a knowledge graph, and contextualizing the query according to the concepts and relationships between the concepts in the knowledge graph.

3. The method of claim 1, further comprising linking the machine learning models to concepts in the knowledge graph corresponding to the machine learning models.

4. The method of claim 1, wherein, The multi-modal unstructured data includes textual data, audio data, video data, and image data.

5. The method of claim 1, wherein, The machine learning models are selected from a library of trained machine learning models.

6. The method of claim 1, wherein, The machine learning models are converted into a structured symbolic form by extracting information associated with the machine learning models.

7. The method of claim 1, wherein, The machine learning models include machine learning classifiers, each trained to classify a particular object.

8. The method of claim 7, wherein, The cropped image resulting from the output of one of the machine learning models is used as input to another of the machine learning models.

9. A system for performing query answering, comprising: a processor; a memory device coupled to the processor; wherein the processor is configured to perform the steps of the method of any of claims 1 to 8.

10. A computer program product comprising a computer readable storage medium containing program instructions readable by a device to cause the device to perform the steps of the method of any of claims 1 to 8.

Images