Computer implementation methods, systems, and computer programs (mapping the application of machine learning models to answer queries according to semantic specifications)

Abstract

To provide a method, system and program for automatically mapping and combining the application of machine learning models to answer queries according to semantic specifications.SOLUTION: A method comprises: receiving a query 502; parsing the query and extracting keywords to contextualize the query 504; based on the keywords, selecting machine learning models that process concepts associated with the keywords 506; sorting the machine learning models according to the contextualization of the query 508; running the machine learning models on multimodal unstructured data according to sorted order such that data resulting from an output of one of the machine learning models is used as input to another one of the machine learning models 510; and outputting a query result based on the run of the machine learning models 512.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] This application generally relates to computers and computer applications, machine learning, automatic question answering, and search engines, and more particularly to automatically mapping and / or combining the application of multiple machine learning models to answer queries according to semantic specifications.

Background Art

[0002] Currently, there is a lack of a mechanism for answering user queries by using symbolic representations and leveraging combinations of multiple machine learning models to extract semantic information from multimodal content. Specifically, current solutions attempt to index all content and store both types of information in a database.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Indexing in this case occurs statically either manually through human annotation or through an exhaustive application of models that can extract concepts regardless of where they are extracted from, which are models available for extracting concepts.

Means for Solving the Problems

[0004] The summary of the present disclosure is provided to assist in understanding computer systems and methods that automatically map and / or combine the application of multiple machine learning models to answer queries, for example, according to semantic specifications, and is not intended to limit the present disclosure or the invention. It should be understood that various aspects and features of the present disclosure can, in some examples, be used advantageously separately, or in other examples, be used advantageously in combination with other aspects and features of the present disclosure. Therefore, changes and modifications can be made to computer systems or their operating methods or both to achieve various effects.

[0005] In one embodiment, the system may include a processor and a memory device coupled to the processor. The processor may be configured to receive queries. The processor may also be configured to parse queries, extract keywords from them, and contextualize the queries. The processor may also be configured to select multiple machine learning models that process concepts related to the keywords based on the keywords. The processor may also be configured to sort the multiple machine learning models according to the contextualization of the queries. The processor may also be configured to run the multiple machine learning models on multimodal data in the sorted order, with data derived from the output of one of the multiple machine learning models being used as input to the other machine learning models. The processor may also be configured to output query results based on the results of running the multiple machine learning models.

[0006] In one embodiment, the computer implementation method may include a step of receiving a query. The method may also include a step of extracting keywords from the query and parsing the query to contextualize it. The method may also include a step of selecting multiple machine learning models to process concepts related to the keywords based on the keywords. The method may also include a step of sorting the multiple machine learning models according to the contextualization of the query. The method may also include a step of running the multiple machine learning models on unstructured, multimodal data in the sorted order, with data derived from the output of one of the multiple machine learning models being used as input to the other machine learning models. The method may also include a step of outputting query results based on the results of running the multiple machine learning models in the sorted order.

[0007] A computer-readable storage medium can also be provided for storing a program of machine-executable instructions for performing one or more of the methods described herein.

[0008] With reference to the attached drawings, further features, structure, and operation of various embodiments are described in detail below. In the drawings, similar reference numerals indicate identical or functionally similar elements. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows the system architecture in one embodiment.

[0010] [Figure 2] This is a flowchart illustrating a method for model introduction and knowledge structuring in one embodiment.

[0011] [Figure 3] This is a flowchart illustrating a method for answering queries in one embodiment.

[0012] [Figure 4] This figure shows an example of a knowledge graph in one embodiment.

[0013] [Figure 5] Another flowchart illustrating the method in one embodiment.

[0014] [Figure 6] This figure shows the components of a system in one embodiment that can automatically map and combine the application of multiple machine learning models to answer queries according to semantic specifications.

[0015] [Figure 7] This is a schematic diagram of an exemplary computer or processing system that can implement the system in one embodiment.

[0016] [Figure 8] This is a diagram of a cloud computing environment in one embodiment.

[0017] [Figure 9] This figure shows a set of functional abstraction layers provided by a cloud computing environment in one embodiment of the present disclosure. [Modes for carrying out the invention]

[0018] In one or more embodiments, artificial intelligence (AI) systems and methods can automatically map, contextualize, and combine the application of multiple machine learning models to answer queries according to semantic specifications. These systems and methods can, for example, contextualize and semantically adjust machine learning (ML) models to relationships between multimodal data and symbolic conceptual labels when answering queries. For example, the system can define the entire ML workflow, including what each model spends and makes through knowledge representations, by describing symbolic neural integration. The system can infer relationships through multimodal data and appropriate models. The system can also process user-specified queries and extract symbolic conceptual content, leveraging other capabilities to map this semantic information to the selection of machine learning models. For example, the system can select, sort, and apply a hierarchy of machine learning models to specific multimodal data and conceptual symbolic representations, dynamically indexing fragments of multimodal data at query time.

[0019] When answering a query, a search or search engine in one embodiment of the system or method or both disclosed herein can consider hierarchy and contextualization in the execution of an information extraction mechanism. For example, this search or search engine can extract meaning or semantic rules from the query and consider the mapping between the functionality of the ML model and the semantic description of the query.

[0020] Some exemplary scenarios are described below. In the first exemplary scenario, consider a user query for an image of a "XYZ magazine" which means that the user is searching for an image of a magazine having the logo "XYZ" shown on the magazine. In one embodiment, the system parses the query and determines that the user only desires images. The system also knows that the user desires two concepts within these images. The system then searches in a contextualized manner for a model that can extract the XYZ logo or word from the image and a model that can identify the magazine within the image. That is, there is a hierarchy in the selection and tuning of the models. For example, the system first executes a magazine identifier on all unprocessed images. Then, with the bounding boxes of the magazines from these images, the system feeds these trimmed images to the next model. This model is an identifier for the XYZ logo or word. The system then presents this result through a dashboard. In this way, the search results can be made more focused and accurate. For example, in the above example, it is possible to remove search results that may include both the XYZ logo and the magazine in a non-contextualized form (e.g., documents that include the word or logo "XYZ" as a separate instance from the image, e.g., those where there is no logo within the magazine image).

[0021] It is also possible to apply the same logic to different fields. For example, in a second exemplary scenario, a geoscientist who is looking for seismic-related images that contain a geological pattern called a mini-basin, specifically only mini-basins with convergent strata, queries the system. In this case, the system selects three models. The first model is for classifying the geological type of the image and filtering and providing only seismic-related images. The second model is a mini-basin classifier. Then, using the bounding box of the mini-basin, the system applies a third model to find the convergent stratum pattern. Then, the system presents the results.

[0022] In a third exemplary scenario that can be considered, the user requests all landscapes of impressionist paintings by British painters. Similarly, the system selects and tunes three models: a painter classifier, an art movement classifier, and a genre classifier.

[0023] The systems or methods, or both, disclosed herein are capable of supporting the introduction of ML models and their association with descriptions of their semantic meanings, and can map semantic descriptions obtained from queries to a contextualized hierarchical set of the best available ML models. These systems or methods, or both, can execute the selected models within query time. In one embodiment, the system or method, or both, can automatically map and combine the application of multiple machine learning models to answer queries according to semantic specifications. The system or method, or both, can describe the semantic rules and possibilities related to multiple machine learning models within a knowledge representation, process queries to extract keywords and meanings and map them to contextualizations of multiple machine learning models, select, sort, and apply selected multiple machine learning models to selected multimodal data and conceptual symbolic representations, and dynamically index multimodal data fragments and multiple machine learning models within query time.

[0024] Figure 1 shows a system architecture in one embodiment capable of implementing a knowledge-oriented, ML-based question-answering (QA) system. The illustrated components include computer implementation components that are implemented and / or run on one or more hardware processors, or coupled to one or more hardware processors. One or more hardware processors may include components such as programmable logic devices, microcontrollers, memory devices, or other hardware components or combinations thereof, which may be configured to perform the respective tasks described herein. Coupled memory devices may be configured to selectively store instructions that can be executed by one or more hardware processors.

[0025] The processor may be a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), other suitable processing components or devices, or one or more of these. The processor may be coupled to a memory device. The memory device may include random-access memory (RAM), read-only memory (ROM), or other memory devices, and may store data or processor instructions or both for implementing various functions relating to the methods or systems or both described herein. The processor may execute computer instructions stored in memory or received from other computer devices or media.

[0026] The dashboard graphical user interface (GUI) 102 may be a user interface module or program capable of supporting interaction between the user and an automated processor or computer. For example, the dashboard GUI 102 may be a browser or a browser-enabled interface, or another computer application program. The user can enter searches or queries via the dashboard GUI 102, and the system can output search results via the dashboard GUI 102.

[0027] The query parser 104 may be a natural language processing module or program that parses queries, such as search queries entered or specified by the user. The query parser 104 can tokenize the search query into tokens for processing. The concept mapper 106 may be a natural language processing module or other program that can analyze the parsed query and determine one or more concepts specified in the query.

[0028] The model selector 112 may be a module or program that searches a database that stores machine learning models, such as a non-symbolic repository 116, and selects machine learning models that relate to or deal with concepts determined by the concept mapper 106. The non-symbolic repository 116 may also store any other non-symbolic or unstructured data, such as documents, images or videos, or combinations thereof.

[0029] The query engine 118 may be a computer module or program that performs a search by applying a combination of related machine learning models determined by the query structure. For example, a search can be performed by running one or more selected machine learning models on unstructured data (e.g., text, images, or videos about text documents, images, or videos, or combinations thereof, etc.). Such unstructured data can also be stored in the non-symbolic repository 116. To address the query, the machine learning models can be executed in a contextualized order. In one embodiment, the query engine is responsible for testing the stored data against selected machine learning models related to the query. Input to the query engine 118 may be a list of related ML models and identified concepts within the query. The query engine 118 can output data that is considered relevant after classification by ML models that can be executed within the model orchestrator 120.

[0030] Model Inspector 122 may be a computer module or program capable of extracting information from an ML model (such as model parameters, inputs and outputs, and other signatures). This information can also be extracted from documentation related to the model.

[0031] The model orchestrator 120 can combine multiple ML models and may be a computer module or program that, for example, executes selected ML models in a specific order, so that, for example, the output of one model can be used as input to another model to reach data to be presented as search results. The model orchestrator 120 can analyze each of the selected models and determine what inputs and outputs each model takes in and produces. Based on this analysis, the model orchestrator 120 can organize or decide which models to execute and in what order, for example, which model's output should be used as input to which model.

[0032] The knowledge graph 110 can be a data structure that stores concepts and relationships between concepts, by including nodes and edges. Nodes can represent concepts, and edges connecting nodes can represent relationships between the nodes to which the edges connect. In one embodiment, edges can be weighted to represent the strength of the relationships between edges. In one embodiment, an existing knowledge graph can be used and augmented with additional information, additional nodes, and edges. In other embodiments, for example, an existing dictionary or ontology that provides concepts and relationships between concepts can be used to construct the knowledge graph. Simply put, an ontology provides descriptions of concepts (e.g., entities) and relationships between concepts. Some examples of knowledge graphs are shown in Figure 4.

[0033] Referring to Figure 1, the knowledge structurer 108 may be a computer module or program that constructs the knowledge graph 110 or enhances the knowledge graph 110 using information. ML models identified from the non-symbolic repository 116 can be linked to concepts or nodes within the knowledge graph 110. In one embodiment, the knowledge structurer 108 is responsible for creating symbolic descriptions of ML models, thereby relating these descriptions to concepts and relationships derived from ontologs within the knowledge graph. For example, the knowledge structurer 108 can create a structure about a model.

[0034] The graph processor 114 can perform functions on graphs such as the knowledge graph 110. For example, it can link information to nodes in the knowledge graph, such as ML models that can function on the concepts represented by the nodes. The graph processor 114 can take in data incorporated into the answer set, structure this data and the characteristics of its source (combinations of ML models used) within the knowledge graph, and create corresponding symbolic descriptions.

[0035] Referring to the exemplary scenario above, a user can enter a search query for images of mugs with the XYZ logo via the dashboard GUI 102. The query parser 104 parses the query. The concept mapper 106 determines concepts such as, for example, "mug" and the "XYZ" logo, and the model selector 112 selects a set of machine learning models from the non-symbolic repository 116 that can perform the query. The model orchestrator 120 contextualizes and combines the execution of multiple machine learning models. For example, it is possible to run a model trained to recognize images of mugs to find the image, or the bounding box of the mug image. The model orchestrator 120 can then provide the cropped image containing only the mug to the next model, for example, the XYZ logo identifier. The output of this model can be used as a search result presented or displayed via, for example, the dashboard GUI 102.

[0036] Similarly, referring to the second exemplary scenario above, the user can use the dashboard 102 to enter a query for searching for earthquake-related images in which convergence layers exist in a minibasin. The query parser 104 parses the query, the concept mapper 106 extracts concepts from the query, and the model selector 112 finds a set of models capable of handling the query from the non-symbolic repository 116a. The model orchestrator 120 contextualizes and combines the execution of the models. For example, the system can find earthquake-related images using an earthquake-related image type classifier. The earthquake-related images can then be fed to a model capable of finding minibasin geological structures. The system can then feed the cropped images containing only the minibasin to the next model, for example, a convergence layer classifier. The results can be presented or displayed via the dashboard GUI 102.

[0037] Similarly, the system shown in Figure 1 can also function to provide search results for the above exemplary scenario answering a query about Impressionist landscape paintings by British painters. It is possible to select different ML models, such as painter classifiers, art movement classifiers, and genre classifiers, and run them in an order determined by the model orchestrator 120 to provide results.

[0038] Figure 2 is a flowchart illustrating a method for model deployment and knowledge structuring in one embodiment. This method can be executed on or implemented by a computer processor, including, for example, a hardware processor. In 202, it is possible to receive models. For example, a user can deploy a model by accessing a dashboard (e.g., 102 in Figure 1). The user can specify a model (e.g., a deep learning model, a neural network, or other ML model) trained to perform a specific task (e.g., recognition of a specific type of image, classification of a specific object or text or a combination thereof). The user can also specify or input information related to the model, such as inputs the model can take in, outputs the model can produce, parameters and other signatures, as well as other information related to the model.

[0039] In another embodiment, 204, the computer processor can automatically extract such information about the introduced model from, for example, available documentation related to the introduced model. For example, a component or module of the Model Inspector (122 in Figure 1) can process the introduced model and extract information from the model, for example, parameters, signatures, or both. Such information may be contained in the documentation related to the model, and the Model Inspector can automatically extract that information.

[0040] In step 206, using the extracted information about the model, a computer processor can automatically create a symbolic description (structured description) of the model and relate this description to the concepts and relationships of the ontology in the knowledge graph. For example, it is possible to construct a knowledge graph that includes concepts and relationships between concepts. Such a knowledge graph can be represented by a data structure of nodes and connecting edges, where nodes represent concepts and edges represent relationships. It is also possible to link or connect the symbolic or structured description of the model to concepts in the knowledge graph. For example, a knowledge structurer (e.g., 108 in Figure 1) can use the extracted information to create a symbolic description of the model and relate this description to one or more concepts in the knowledge graph (e.g., 110 in Figure 1).

[0041] In version 208, it is possible to present modified knowledge graphs. For example, a dashboard GUI (e.g., 102 in Figure 1) can display or present modified knowledge graphs, enabling ontological alignment through user interaction, user curation, and learning. For instance, users can further edit the knowledge graph by providing feedback on the modified knowledge graph.

[0042] At step 210, it is determined whether to introduce another model. If so, the logic proceeds to step 202; otherwise, the logic can terminate or return to its own calling module.

[0043] Figure 3 is a flowchart illustrating a method for answering queries in one embodiment. This method can be executed on or implemented by a computer processor, for example, a hardware processor. At 302, a query is received. For example, a user can specify a query by accessing a dashboard GUI (e.g., 102 in Figure 1). At 304, the computer processor extracts the keywords and overall structure of the query, for example, using natural language processing techniques. For example, a query parser (e.g., 104 in Figure 1) can extract the keywords and overall structure of the query.

[0044] In step 306, the computer processor maps the extracted keywords to concepts and relationships within the knowledge graph and uses the extracted structure to construct the corresponding queries in knowledge graph terminology. For example, the concept mapper (106 in Figure 1) maps the extracted keywords to concepts and relationships within the knowledge graph and uses the extracted structure to construct the corresponding queries in knowledge graph terminology.

[0045] In step 308, the computer processor relates ML models (e.g., those obtained from a repository) as a decision-making procedure for specific concepts and relationships used in a query. For example, a model selector (e.g., 112 in Figure 1) relates ML models (e.g., those obtained from a non-symbolic repository (e.g., 116 in Figure 1)) as a decision-making procedure for specific concepts and relationships used in a query created by a concept mapper (e.g., 106 in Figure 1).

[0046] In step 310, a computer processor takes in a query using an associated ML model and evaluates it via a knowledge graph. The evaluation process may include zero or more evaluation stages. For example, a query engine (e.g., 118 in Figure 1) takes in a query using an associated ML model and evaluates the query via a knowledge graph (e.g., 110 in Figure 1).

[0047] In one embodiment, query evaluation of 310 may include graph pattern patching along with connectivity testing. The query itself can be represented by a graph with constraints in addition to "gaps". In one embodiment, the task of the query engine is to find nodes and edges in the target graph that fill these gaps while satisfying the constraints. Specifically, the query engine may attempt to find all subgraphs in the target graph that are identical (or, in some cases, identical) to the query graph.

[0048] For example, to evaluate the query "select x where {x instanceOf Person}", the query engine first transforms it into a graph of the form "(x) -instanceOf->(Person)", which has two nodes, (x) and (Person), and an edge labeled "instanceOf(instance of)" connecting the former to the latter. Next, the query engine searches the target graph for all nodes x from x to the "Person" node that have an edge labeled "instanceOf(instance of)". Any x that satisfies this test is the result of the query.

[0049] More complex queries can include connectivity tests. For example, "select x where {x likes+ B}" means not only all x for which the "likes" edge exists between x and "B", but also all x for which there is a non-zero path of a "subclass" edge connecting x and "B", due to the plus (+) operator in the query. This plus (+) operator introduces a connectivity test. In this case, the query is looking for all x that like B, or all x that like someone who likes B, or all x that like someone who likes someone who likes B, and so on.

[0050] In one embodiment, an ML model can be used to determine whether the constraints of a query are met. For example, suppose there is an ML classifier that, given an image, can determine whether that image depicts a person. It is also assumed that it is desirable to evaluate the query "select x where {x instanceOf Image, x depicts Person}". In this case, the query engine would attempt to find all x in the target graph such that there is an "instanceOf" edge from x to "Image" and "depicts" from x to "Person". Here, there may be an image I in the target graph that has not yet been analyzed. All that is known about it is that it is an image. Therefore, the task here is to find all images that have a given constraint ("depicts a person"), and given an image, there is a model M in the ML library that can determine whether this constraint holds, so the query engine can use M to determine whether, given an unanalyzed image I, there should be an edge from I to "Person" that "depicts". If M answers "yes" given model I, in one embodiment the query engine proceeds to add this edge to the graph (along with its origin, i.e., the fact that the edge derives from model M's evaluation) and adds I to the result set.

[0051] In step 312, it is determined whether there are further evaluation stages. For example, if there are additional question components to answer about the query, or if there are additional related ML models to run, it can be determined that there are further evaluation stages remaining. For example, it is possible to test conjunctions in the query and ML compositions. As an example of a conjunction, consider the query "select x where {x instanceOf Image, x depicts Person, x depicts Cat}" (select x {x is an instance of an image, x depicts a person, x depicts a cat}). That is, it is desired to find all images x that depict both a person and a cat. Assume there are models M1 and M2 that can determine whether a person appears in an image and whether a cat appears in an image, respectively. When considering an unanalyzed image I, there are two constraints that the image must satisfy in order to be included in the result set, which can be tested by models M1 and M2, respectively. The query engine will include I in the result set if both M1 and M2 respond "yes" when applied to I, i.e., if M1(I)&M2(I) is true. Here, "&" is the conjunction operator.

[0052] As an example of composition, consider the query "select x where {x instanceOf Image, x depicts Animal, x depicts Zebra}" and assume that there are two models, M1 and M2, where M1 can determine whether an image is an image of an animal, and M2 can determine whether an image of an animal is actually an image of a zebra. An unanalyzed image I will satisfy this query if both models answer "yes", but M2 is only applicable to images of animals (unanalyzed image I is not a suitable input for M2), so the order in which these models are applied is important. In one embodiment, the order of evaluation is determined by the input / output constraints of the models themselves, which are described in the knowledge graph. In this case, when considering image I, the query engine first applies M1, and only applies M2 if M1 answers "yes".

[0053] More specifically, in one embodiment, the query engine can apply the model in a specific order determined by the model's signature. That is, this method can represent M1 as a partial function from "Image" to "Animal" (from general images to images of animals) and M2 as a partial function from "Animal" to "Zebra" (from images of animals to images of zebras). In notation, this is M1:Image->Animal and M2:Animal->Zebra. Given an image of type I, the query engine works in reverse when considering the constraint "depicts a zebra". The query engine knows that M2 can answer the query, but only if I is an image of an animal (which M1 can answer directly). Therefore, if M2(M1(I)) is defined, that is, if the composition (M2*M1)(I) is defined, the query engine will include I in the result set. By definition of the function composition, the latter is undefined if M1(I) is undefined.

[0054] If there is a further evaluation stage, in 314, the computer processor applies a combination of related ML models (determined by the query structure) to the data in the repository and determines whether the considered data should be part of the result set. For example, the query engine (e.g., 118 in Figure 1) uses a model orchestrator (e.g., 120 in Figure 1) to apply a combination of related ML models (determined by the query structure) to the data in a non-symbolic repository (e.g., 116 in Figure 1) and determines whether the considered data should be part of the result set.

[0055] In 316, a computer processor takes in the data incorporated into the result set and adds this data, along with its source (the combination of ML models used to classify the data), to a knowledge graph to create a corresponding symbolic description. For example, a graph processor (e.g., 114 in Figure 1) takes in the data incorporated into the result set and adds this data, along with its source (the combination of ML models used to classify the data), to a knowledge graph (e.g., 110 in Figure 1) to create a corresponding symbolic description.

[0056] In step 312, if no other evaluations to be performed exist, in step 318, the computer processor presents the query results (final result set) to enable further ontology alignment, user curation, and learning through user interaction. For example, the user can provide feedback on the query results. Based on the feedback, one or more ML models and the knowledge graph can be modified. The query results can be presented or displayed, for example, via a dashboard GUI (e.g., 102 in Figure 1).

[0057] In 320, if there are other queries to process, the logic proceeds to 302. Otherwise, the logic can terminate or return to the calling process.

[0058] One approach involves a semantically guided combination of dynamic machine learning models that execute query answers through a combination of machine learning models applied incrementally (in query time) to fragments of data described within a contextualized knowledge graph.

[0059] Figure 4 shows an example of a knowledge graph in one embodiment. Nodes in the knowledge graph 400 can represent concepts, and edges can represent relationships. For example, specific painter nodes 402 and 404 (e.g., containing a suitable name or identifier for a particular painter) may have a relationship (have nationality) with specific country nodes 406 and 408. Specific painter nodes 402 and 404 may also have a relationship (have occupation) with the occupation node "painter" 414. Specific painting node 410 (e.g., referable by name or title or other identifier, and potentially containing information such as genre or type and other information) may have a relationship (e.g., being a subconcept of) with the subject node "painting" 412. This particular painting may also have a relationship (e.g., having a creator) with a specific painter node (e.g., 402 or 404). In one embodiment, the knowledge graph 400 can be updated based on a search. For example, through a search, it may be discovered that "Painter A" 402 is the creator of "Painting 1" 410. In this scenario, it is possible to update knowledge graph 400 to include the relationship "Has Creator" between "Painter A" 402 and "Painting 1" 410.

[0060] Figure 5 is another flowchart illustrating the method in one embodiment. This method can be executed on, for example, a computer processor including a hardware processor, or implemented by such a computer processor. At 502, it is possible to receive queries. For example, a user via a user interface or graphical user interface can specify a question or query to the computer, for example, seeking an answer or response to the query. At 504, using natural language processing techniques, the processor or computer processor can parse the query and extract keywords used in the query. The processor also contextualizes the query by semantically analyzing its structure. In one embodiment, it is possible to map the keywords to concepts in a knowledge graph and contextualize the query according to the relationships between concepts in the knowledge graph.

[0061] In 506, based on the keywords, the processor searches for, identifies, or selects machine learning models that can handle or process concepts related to the keywords. For example, it can identify or select machine learning models that handle concepts related to one keyword in the query, and other machine learning models that handle other concepts related to other keywords in the query. In one embodiment, a machine learning model is selected from a repository containing trained machine learning models. Examples of machine learning models may include neural networks, deep learning networks, and other unsupervised, semi-supervised, and / or supervised machine learning models. In one embodiment, it is possible to link or associate machine learning models with each concept in the knowledge graph. For example, in one embodiment, the machine learning models may include machine learning classifiers, each machine learning model being trained to classify a particular object. For example, each model may be capable of performing different classifications.

[0062] In 508, a processor or computer processor sorts machine learning models according to a contextualized query or context. For example, if the query involves searching for images of items that display a specific logo within an item, and a machine learning model that classifies images of items and another machine learning model that classifies images of logos are selected, the processor may decide to sort the machine learning models in the order of machine learning models that classify images, and then machine learning models that classify images of logos.

[0063] In 510, a processor or computer processor may run machine learning models on unstructured, multimodal data in a sorted order, and use data derived from the output of one of the machine learning models as input to another of the machine learning models. Examples of multimodal data may include, but are not limited to, text data, audio data, video data, and image data. For example, the first of the machine learning models may take unstructured, multimodal data as input and classify that data. The first of the machine learning models may then crop an image that has been classified as containing the item or subject being searched for, and feed that input to the second of the machine learning models to further classify it in order to obtain a desired result. In one embodiment, composition is used in any case in which the output of one model works as input for other models, such as when one model crops a relevant portion of an image, and that portion is then fed into a second or other model. For example, in the example above involving images of animals and zebras, instead of simply answering yes or no when given an image of an animal, model M1 can create a new image J by cropping the portion of the original image I in which the animal appears and answering yes for J. The new image J is then fed to M2, and if it is an image of a zebra, J is included in the result set. This can also be described in reverse, such as I not being an image of a zebra but potentially containing an image of a zebra. One way to find a suitable input for M2 is to use model M1 to consider a sub-image of I. In 512, the search results are output based on the execution of the machine learning models in a sorted order.

[0064] Figure 6 shows the components of a system in one embodiment that can automatically map and combine the application of multiple machine learning models to an answer query according to semantic specifications. One or more hardware processors 602, such as a central processing unit (CPU), a graphics processing unit (GPU), and / or a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and / or other processors, are coupled to a memory device 604 and can execute query answers by using machine learning models according to the semantic rules of the query. The memory device 604 may include random access memory (RAM), read-only memory (ROM), or other memory devices and can store data or processor instructions or both for implementing various functions related to the method or system or both described herein. One or more processors 602 can execute computer instructions stored in the memory 604 or received from other computer devices or media. The memory device 604 may store instructions or data or both about the functions of one or more hardware processors 602, and may include operating systems and other programs for instructions or data or both. One or more hardware processors 602 can receive inputs, for example, unstructured data, or multimodal data, or inputs containing unstructured multimodal data, for one or more machine learning models to be classified in order to answer a query. In one embodiment, unstructured multimodal data, or non-symbolic multimodal data, can be stored in a storage device 606 or received from a remote device via a network interface 608, and the data can be temporarily loaded into a memory device 604 for classification. One or more machine learning models can be stored on the memory device 604, for example, for execution by one or more hardware processors 602.One or more hardware processors 602 can be coupled to interface devices such as a network interface 608 for communicating with a remote system over a network, and an input / output interface 610 for communicating with an input device such as a keyboard or mouse or both, or an output device or both.

[0065] Figure 7 is a schematic diagram of an exemplary computer or processing system that can implement the system in one embodiment. This computer system is merely one example of a suitable processing system and is not intended to imply any limitation on the scope of use or functionality of the embodiments of the methodology described herein. The illustrated processing system may operate with a great many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, computing environments or computing configurations or combinations thereof that may be suitable for use with the processing system shown in Figure 7 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of these systems or devices.

[0066] This computer system can be described in the general context of computer system executable instructions executed by the computer system, such as program modules. Generally, program modules may include routines, programs, objects, components, logic, and data structures that perform a specific task or implement a specific abstract data type. Computer systems can be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices linked over a communication network. In a distributed cloud computing environment, program modules can reside in both local computer system storage media, including memory storage devices, and remote computer system storage media.

[0067] The components of the computer system may include, but are not limited to, one or more processors or processing units 12, system memory 16, and a bus 14 that connects various system components, including the system memory 16, to the processor 12. The processor 12 may include a module 30 that performs the methods described herein. The module 30 can be programmed within the integrated circuit of the processor 12, or it can be loaded from memory 16, storage devices 18, or a network 24, or a combination thereof.

[0068] Bus 14 may represent one or more types of bus structures from among various types of bus structures, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor buses or local buses using any of the various bus architectures. Examples of such architectures include, but are not limited to, industry standard architecture (ISA) buses, microchannel architecture (MCA) buses, enhanced ISA (EISA) buses, video electronics standards association (VESA) local buses, and peripheral component interconnect (PCI) buses.

[0069] A computer system may include various computer system-readable media. These media may be any available media accessible to the computer system, and may include both volatile and non-volatile media, as well as removable and non-removable media.

[0070] System memory 16 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM), cache memory, or both. The computer system may further include other removable / non-removable computer system storage media, volatile / non-volatile computer system storage media. For example, a storage system 18 may be provided for reading from and writing to a non-removable non-volatile magnetic medium (e.g., a “hard drive”). Not shown, a magnetic disk drive may be provided for reading from and writing to a removable non-volatile magnetic disk (e.g., a “floppy disk”), and an optical disk drive may be provided for reading from or writing to a removable non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media. In such examples, each may be connected to the bus 14 by one or more data medium interfaces.

[0071] The computer system may also communicate with one or more external devices 26, such as a keyboard, pointing device, and display 28, and with one or more devices that enable a user to interact with the computer system, or any device that enables the computer system or both to communicate with one or more other computing devices (e.g., a network card, modem, etc.). Such communication may occur via an input / output (I / O) interface 20.

[0072] Furthermore, the computer system can communicate with one or more networks 24, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via a network adapter 22. As shown in the figure, the network adapter 22 communicates with other components of the computer system via a bus 14. It should be understood that other hardware components, software components, or both, which are not shown, can be used in conjunction with the computer system. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0073] This disclosure may include a description of cloud computing, but it should be understood in advance that the implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment that is currently known or may be developed in the future. Cloud computing is a service delivery model that enables on-demand, convenient network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0074] The following are its characteristics.

[0075] On-demand self-service: Cloud consumers can unilaterally provision computing functions such as server time and network storage automatically as needed, without the need for human interaction with the service provider.

[0076] Broad network access: Functionality is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin-client or thick-client platforms (e.g., mobile phones, laptops, and PDAs).

[0077] Resource pooling: To serve multiple consumers using a multi-tenant model, a provider pools its computing resources, with various physical and virtual resources dynamically allocated and reallocated as needed. Consumers generally cannot control or have knowledge of the exact location of the resources provided, but they can specify the location at a higher level of abstraction (e.g., country, state, or data center), thus exhibiting a kind of location independence.

[0078] Rapid Scalability: In some cases, features can be automatically and quickly provisioned to scale out, and quickly released to scale in. For consumers, the features available for provisioning are often unlimited and can be purchased anytime, in any quantity.

[0079] Service Measurement: Cloud systems, at some level of abstraction, automatically control and optimize resource usage by leveraging metric capabilities appropriate for the type of service (e.g., storage, processing, bandwidth, and active user accounts). Monitoring, controlling, and reporting resource usage can provide transparency to both service providers and consumers.

[0080] The following is the service model.

[0081] Software as a Service (SaaS): The functionality provided to consumers utilizes the provider's applications running on cloud infrastructure. These applications are accessible from various client devices via thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or individual application functionalities, with the possible exception of limited, user-specific application configuration settings.

[0082] Platform as a Service (PaaS): The functionality provided to consumers allows them to deploy consumer-created or acquired applications, written using programming languages ​​and tools supported by the provider, onto the cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but they control the deployed applications and, in some cases, the applications that act as hosts for the environment configuration.

[0083] Infrastructure as a Service (IaaS): This service provides consumers with the provisioning of computing resources, including processing, storage, networking, and other underlying computing resources that enable them to deploy and run any software they choose, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do control the operating system, storage, and deployed applications, and, in some cases, have limited control over selected networking components (e.g., host firewalls).

[0084] The following is the deployment model.

[0085] Private Cloud: A cloud infrastructure that operates solely for a specific organization. The cloud infrastructure can be managed by that organization or a third party and can reside on-premises or off-premises.

[0086] Community Cloud: Cloud infrastructure is shared by various organizations to support specific communities that share common interests (e.g., mission, security requirements, policies, and compliance considerations). The cloud infrastructure can be managed by those organizations or third parties and can reside on-premises or off-premises.

[0087] Public cloud: Cloud infrastructure is made available to the general public or large industry groups and is owned by organizations that sell cloud services.

[0088] Hybrid cloud: A combination of two or more clouds (private, community, or public) where the cloud infrastructure remains a distinct entity, but is linked by standardized or proprietary technologies that bring about data and application portability (e.g., cloud bursting for load balancing across clouds).

[0089] Cloud computing environments are services oriented towards statelessness, loose coupling, modularity, and semantic interoperability. The infrastructure, including a network of interconnected nodes, is central to cloud computing.

[0090] Referring now to Figure 8, an exemplary cloud computing environment 50 is shown. As illustrated, the cloud computing environment 50 includes one or more cloud computing nodes 10 that can communicate with local computing devices used by cloud consumers (e.g., personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, or automotive computer systems 54N, or a combination thereof). The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) within one or more networks, such as private clouds, community clouds, or hybrid clouds, or a combination thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platforms, or software, or all of them, as a service, without requiring cloud consumers to maintain resources on their local computing devices. The types of computing devices 54A through 54N shown in Figure 8 are intended to be illustrative only, and it should be understood that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of computerized device via any type of network or network addressable connection or both (e.g., using a web browser).

[0091] Next, referring to Figure 9, a set of functional abstraction layers provided by the cloud computing environment 50 (Figure 8) is shown. It should be understood that the components, layers, and functionalities shown in Figure 9 are for illustrative purposes only, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functionalities are provided:

[0092] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include a mainframe 61, RISC (Reduced Instruction Set Computer) architecture-based servers 62, 63, blade servers 64, storage devices 65, and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0093] The virtualization layer 70 provides an abstraction layer. From the abstraction layer, examples of the following virtual entities may be provided: virtual servers 71, virtual storage 72, virtual networks 73 including virtual private networks, virtual applications and operating systems 74, and virtual clients 75.

[0094] For example, the management layer 80 can provide the functions described below. Resource provisioning 81 dynamically procures computing resources and other resources used to perform tasks within the cloud computing environment. Measurement and pricing 82 tracks costs as resources are used within the cloud computing environment and bills or creates invoices for the consumption of these resources. For example, these resources may include application software licenses. Security verifies the identity of cloud consumers and tasks, and protects data and other resources. User portal 83 gives consumers and system administrators access to the cloud computing environment. Service level management 84 allocates and manages cloud computing resources to ensure that the required service levels are met. Service level agreement (SLA) planning and fulfillment 85 proactively coordinates and procures cloud computing resources that are expected to be needed in the future in accordance with SLAs.

[0095] The workload layer 90 provides examples of functions that can utilize a cloud computing environment for their functions. Examples of workloads and functions that can be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analysis processing 94, transaction processing 95, and query response processing 96.

[0096] The present invention may be a system, method, or computer program product, or all of them, in an integration at any possible level of technical detail. The computer program product may include a computer-readable storage medium (or multiple computer-readable storage media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0097] Computer-readable storage media can be tangible devices capable of holding and storing instructions used by instruction execution devices. Computer-readable storage media may be, but are not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. A non-exclusive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punched cards or grooved raised structures on which instructions are recorded, and any suitable combination thereof. In this specification, computer-readable storage media themselves are not considered to be radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or transient signals such as electrical signals transmitted through wires.

[0098] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing device / processing device, or downloaded to an external computer or external storage device via a network such as the Internet, a local area network, a wide area network, or a wireless network, or all of these. The network may include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or all of these. A network adapter card or network interface within each computing device / processing device receives computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in the computer-readable storage medium within the respective computing device / processing device.

[0099] The computer-readable program instructions for performing the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk® or C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions can run as a standalone software package, either entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or a connection to an external computer can be established (for example, via the Internet using an Internet Service Provider). In some embodiments, to carry out aspects of the present invention, an electronic circuit including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) can execute a computer-readable program instruction by personalizing the electronic circuit using state information of the computer-readable program instruction.

[0100] Aspects of the present invention are described herein with reference to flowcharts or block diagrams, or both, of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart or block diagram, or both, and any combination of blocks in a flowchart or block diagram, or both, can be implemented by computer-readable program instructions.

[0101] These computer-readable program instructions can be provided to a computer processor or other programmable data processing device to create a machine, thereby creating a means for instructions executed via the computer processor or other programmable data processing device to implement functions / operations defined in one or more blocks of a flowchart or block diagram, or both. These computer-readable program instructions can also be stored in a computer-readable storage medium capable of instructing a computer, a programmable data processing device, or other device, or all of these, to function in a particular manner, thereby including a product in which the computer-readable storage medium storing the instructions contains instructions that implement modes of functions / operations defined in one or more blocks of a flowchart or block diagram, or both.

[0102] Computer-readable program instructions can also be loaded into a computer, other programmable data processing device, or other device to create a computer implementation process by executing a series of operational steps on the computer, other programmable device, or other device, thereby enabling the instructions executed on the computer, other programmable device, or other device to implement functions / operations defined by one or more blocks in a flowchart or block diagram, or both.

[0103] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown in a block may occur in an order other than that shown in the figure. For example, in practice, two consecutively shown blocks may be implemented as a single stage, executed simultaneously, executed substantially simultaneously, executed partially or completely in overlapping time, or, depending on the functions they contain, executed in reverse order. It should also be noted that each block in a block diagram or flowchart, or both thereof, and any combination of blocks in a block diagram or flowchart, or both thereof, can be implemented by a special-purpose hardware-based system that performs a specified function or operation, or executes a combination of special-purpose hardware instructions and computer instructions.

[0104] The terms used herein are intended to describe only specific embodiments and are not intended to limit the invention. In this specification, unless the context explicitly indicates otherwise, the singular forms “a,” “an,” and “the” are intended to include the plural forms. In this specification, unless the context explicitly or explicitly indicates otherwise, the word “or” is an inclusive sign and may mean “and / or.” The words “comprise,” “comprises,” “comprising,” “include,” “includes,” “including,” or “having,” or any combination thereof, when used herein, may specify the presence of a described feature, complete, stage, operation, element, or component, or a combination thereof, but will not exclude the presence or addition of one or more other features, complete, stage, operation, element, component, or group thereof, or combination thereof. In this specification, the phrase “in an embodiment” does not necessarily refer to the same embodiment, but may refer to the same embodiment. In this specification, the phrase "in one embodiment" does not necessarily refer to the same embodiment, but may refer to the same embodiment. In this specification, the phrase "in another embodiment" does not necessarily refer to a different embodiment, but may refer to a different embodiment. Furthermore, embodiments or components of embodiments, or both, can be freely combined with each other, except where they are incompatible.

[0105] Where the following claims include means or step plus functional elements, the corresponding structures, materials, actions, and equivalents of all such elements are intended to include any structures, materials, or actions for performing the function in combination with other claimed elements specifically claimed. The description of the present invention has been presented for illustrative and explanatory purposes, but is not intended to be exclusive or limited to the disclosed forms of the invention. Many modifications and variations that do not depart from the scope and spirit of the invention will be apparent to those skilled in the art. The embodiments have been selected and described to best illustrate the principles and practical applications of the present invention and to enable those other skilled in the art to understand the invention in terms of various embodiments with various modifications suitable for specific intended uses.

Claims

1. A method by which a computer performs an action, The stage where the computer receives a query, The computer extracts keywords from the query and parses the query in order to contextualize it. A step in which a computer selects multiple machine learning models to process concepts related to the keyword, wherein each different machine learning model handles different concepts related to different keywords in the query. The steps include: a computer sorting the plurality of machine learning models according to the contextualization of the query; A step in which a computer runs the plurality of machine learning models on unstructured, multimodal data in a sorted order, wherein data derived from the output of one of the plurality of machine learning models is used as input to the other machine learning models among the plurality of machine learning models. The computer outputs query results based on the results from the step of running the multiple machine learning models in the sorted order. A method for providing this.

2. The aforementioned keywords are mapped to concepts within the knowledge graph, and the queries are contextualized according to the relationships between the aforementioned concepts within the knowledge graph. The method according to claim 1.

3. The computer further includes a step in which it links the multiple machine learning models to their respective concepts in the knowledge graph. The method according to claim 1 or 2.

4. The aforementioned multimodal data includes text data, audio data, video data, and image data. The method according to claim 1 or 2.

5. The aforementioned multiple machine learning models are selected from a repository of trained machine learning models. The method according to claim 1 or 2.

6. By extracting information related to the aforementioned multiple machine learning models, the aforementioned multiple machine learning models are transformed into a structured symbolic form. The method according to claim 1 or 2.

7. The aforementioned plurality of machine learning models include machine learning classifiers, each of which is trained to classify a specific object. The method according to claim 1 or 2.

8. A cropped image derived from the output of one of the multiple machine learning models is used as the input to the other machine learning models among the multiple machine learning models. The method according to claim 7.

9. Processor and The memory device coupled to the aforementioned processor Equipped with, The aforementioned processor, Receiving queries and Extracting keywords from the aforementioned query, parsing the query in order to contextualize it, The process involves selecting multiple machine learning models that process concepts related to the aforementioned keywords, wherein each different machine learning model handles different concepts related to different keywords in the query. Sort the multiple machine learning models according to the contextualization of the query, The process involves executing the aforementioned multiple machine learning models on multimodal data in a sorted order, wherein data derived from the output of one of the aforementioned machine learning models is used as input to the other machine learning models among the aforementioned models. Based on the results obtained from running the aforementioned multiple machine learning models, query results are output. It is configured to do, system.

10. The aforementioned keywords are mapped to concepts within the knowledge graph, and the queries are contextualized according to the relationships between the aforementioned concepts within the knowledge graph. The system according to claim 9.

11. The processor is further configured to link the multiple machine learning models to their respective concepts in the knowledge graph. The system according to claim 9 or 10.

12. The aforementioned multimodal data includes text data, audio data, video data, and image data. The system according to claim 9 or 10.

13. The aforementioned multiple machine learning models are selected from a repository of trained machine learning models. The system according to claim 9 or 10.

14. By extracting information related to the aforementioned multiple machine learning models, the aforementioned multiple machine learning models are transformed into a structured symbolic form. The system according to claim 9 or 10.

15. The aforementioned plurality of machine learning models include machine learning classifiers, each of which is trained to classify a specific object. The system according to claim 9 or 10.

16. A cropped image derived from the output of one of the multiple machine learning models is used as the input to the other machine learning models among the multiple machine learning models. The system according to claim 9 or 10.

17. On the computer, The procedure for receiving queries, A procedure for extracting keywords from the aforementioned query and parsing the query in order to contextualize it, A procedure for selecting multiple machine learning models that process concepts related to the aforementioned keywords, wherein each different machine learning model handles different concepts related to different keywords in the query, A procedure for sorting the multiple machine learning models according to the contextualization of the query, A procedure for running the multiple machine learning models on multimodal data in a sorted order, wherein data derived from the output of one of the multiple machine learning models is used as input to the other machine learning models. A procedure for outputting query results based on the results obtained from running the aforementioned multiple machine learning models. A computer program designed to execute something.

18. The aforementioned keywords are mapped to concepts within the knowledge graph, and the queries are contextualized according to the relationships between the aforementioned concepts within the knowledge graph. The computer program according to claim 17.

19. To the aforementioned computer, Further steps are taken to link the aforementioned multiple machine learning models to their respective concepts within the knowledge graph. The computer program according to claim 17 or 18.

20. The aforementioned multimodal data includes text data, audio data, video data, and image data. The computer program according to claim 17 or 18.

Images