Collaborative extended language model for capturing and utilizing domain expertise

Abstract

To provide a system, method and memory for capturing and utilizing expert knowledge in artificial intelligence.SOLUTION: A system includes a knowledge capture module 1210 for extracting expert knowledge from subject matter experts in a conversational format, a knowledge management module 1220 for cataloging and summarizing the knowledge, and a digital subject matter expert (dSME) module 1230 for guiding a user in building AI models to ingest the knowledge. The knowledge capture module interacts with a user and selects the appropriate dSME module that is relevant to the user request. The system attempts to answer the user request based on the dSME module. If the dSME module lacks knowledge to solve the problem, the system uses a set of tools of internet based search engine to solve the problem.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present disclosure relates generally to artificial intelligence and machine learning techniques, and more specifically to capturing and utilizing expert knowledge in the development of artificial intelligence-based systems. [Background technology]

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 510,335, filed June 26, 2023, which is incorporated by reference in its entirety.

[0003] Artificial intelligence (AI) techniques are beneficial to several industrial systems. For example, machine learning based models are used to make predictions that are used in industrial processes. There are several challenges in developing artificial intelligence techniques for industrial systems. For example, developing AI solutions can be time consuming and requires huge amounts of data and expert knowledge. Unavailability of large data in physical space and lack of communication between data scientists and subject matter experts can lead to long development times for AI solutions. AI solutions based on incomplete information can provide inaccurate and incomplete results. Summary of the Invention

[0004] The system answers a domain-specific problem specified by a user by using a natural language request. The system receives a natural language request to answer a question from one of a plurality of domains. The natural language request includes a domain-specific problem. The system determines whether any of a plurality of domain-specific models are configured to answer the domain-specific problem. Each of the plurality of domain-specific models stores a domain-specific knowledge base. If the system determines that at least one of the plurality of domain-specific models is configured to answer the domain-specific problem specified in the natural language request, the system selects a domain-specific model from the plurality of domain-specific models to answer the domain-specific problem. The system executes the domain-specific model to answer the domain-specific problem. If the system determines that none of the plurality of domain-specific models is capable of answering the domain-specific problem, the system selects a software tool from a plurality of software tools to answer the domain-specific problem. The system sends the request to the software tool to answer the domain-specific problem.

[0005] The embodiment performs the steps of the method disclosed herein. The embodiment includes a computer readable medium storing instructions for performing the steps of the above method. The embodiment includes a computer system including one or more computer processors and a computer readable medium storing instructions for performing the steps of the above method.

[0006] The disclosed aspects have other advantages and features that will become more readily apparent from the detailed description, the appended claims, and the accompanying drawings (or figures), a brief introduction of which follows.

[0007] The features and advantages described herein are not all-inclusive, and many additional features and advantages will become apparent to those skilled in the art, especially in view of the drawings, specification, and claims. Moreover, it should be noted that the style used herein has been selected primarily for readability and instructional purposes, and may not have been selected to delineate or limit the disclosed subject matter. [Brief description of the drawings]

[0008] [Figure 1] 1 illustrates an overall system environment of a knowledge-based AI system according to an embodiment of the present invention. [Diagram 2] 1 illustrates a system architecture of a Knowledge First system according to an embodiment. [Diagram 3] FIG. 2 illustrates an overall process for making predictions according to an embodiment of the present invention. [Figure 4] 1 illustrates a development system for use in building an AI system according to an embodiment. [Diagram 5] FIG. 1 illustrates the overall architecture of a knowledge-based AI system according to an embodiment. [Figure 6] FIG. 1 illustrates an overall process for making predictions using a knowledge-based AI system according to an embodiment. [Figure 7] FIG. 1 illustrates the use of various tools for use by a knowledge-based AI system 150 according to an embodiment. [Figure 8] FIG. 1 illustrates the use of a knowledge-based AI system in relation to an application in accordance with various embodiments. [Figure 9] FIG. 1 illustrates the use of a knowledge-based AI system in relation to an application in accordance with various embodiments. [Figure 10] FIG. 1 illustrates the use of a knowledge-based AI system in relation to an application in accordance with various embodiments. [Figure 11] FIG. 1 illustrates the use of a knowledge-based AI system in relation to an application in accordance with various embodiments. [Figure 12] 1 illustrates an architecture of a system for developing AI solutions according to an embodiment. [Figure 13] FIG. 2 illustrates interactions between various components of a system according to an embodiment. [Figure 14] FIG. 1 illustrates four Knowledge First (K1st) architectures according to various embodiments. [Figure 15] 1 shows a flowchart illustrating a process for answering domain-specific natural language requests according to an embodiment. [Figure 16] FIG. 1 is a high-level block diagram illustrating an exemplary system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A system according to an embodiment captures and utilizes expert knowledge in AI development for applications such as industrial systems. A user interacts with a chatbot (referred to herein as an assistant chatbot), which interacts with various tools to solve the user's problem. For example, a user may ask a natural language question to the assistant chatbot that provides information about a problem (e.g., of an equipment) requesting the assistant chatbot 1320 to diagnose the problem. If the assistant chatbot has the appropriate information / knowledge, the assistant chatbot answers the question. If not, the assistant chatbot determines which tool (or component) to interact with to get the answer. The assistant chatbot interacts with one or more tools to solve the user's problem, such as various specialized LLMs, web search tools 1340, mathematical processing tools 1350 (e.g., Wolfram math tools), problem solving LLM 1330 to generate a sequence of steps to solve the problem, etc.

[0010] System Environment A system according to an embodiment implements a knowledge-first architecture that allows for the incorporation of expert, e.g., domain expert, knowledge into the development and use of an AI system. The system is referred to as a knowledge-based AI system, or alternatively, a knowledge-first system. The AI ​​system includes one or more predictive nodes, each of which represents a computational system that receives input data and makes one or more predictions that may be used in the system function. For example, the input data may be sensor data generated by an industrial system, and the predictions may indicate whether the industrial system is faulty.

[0011] According to an embodiment, the knowledge-based AI system comprises a prediction unit that uses the knowledge model to provide training labels for the generalized ML model and to provide a predicted output for the functional system even in the absence of a fully trained ML model. The system also includes an ensemble model that aggregates the outputs of both the expert-made knowledge model and the generalized (ML) model and outputs a final decision. The ensemble model can combine these outputs in various ways. According to an embodiment, the ensemble model combines the outputs using a logical AND or OR operation between the output of the generalized ML model and the output of the knowledge model. According to another embodiment, the ensemble model checks the model accuracy of the ML model and prioritizes the knowledge model output if the accuracy of the ML model is low. According to an embodiment, the ensemble model is implemented as an ML model and learns to optimally use both ML and knowledge outputs to generate a final decision for the system operation.

[0012] Knowledge models also come in different forms and can be adapted to different use cases. The simplest implementation is a logical operation on the input data, outputting either a Boolean classification or a more detailed category label. For predictive maintenance and failure prediction use cases, unsupervised anomaly detection is done on the input dataset before passing the data of anomalous points to the oracle. In this case, the knowledge model incorporates experts, people with years of experience maintaining the system in question. The expert user specifies rules related to the original sensor variables, for example, "If sensor A > threshold A and sensor B < threshold B then output error C". In this way, the knowledge model classifies anomalous data points as a certain type of error. However, as data accumulates and is labeled by the knowledge model, the associated ML models become more accurate and functional, and both models provide valuable outputs, with the ensemble model utilizing insights from both to draw the final conclusion.

[0013] Figure 1 illustrates an overall system environment of a knowledge-based AI system according to an embodiment of the present invention. The overall system environment includes one or more devices 130, a knowledge-based artificial intelligence system or knowledge-based AI system 150, and a network 110. Other embodiments may use more, fewer, or different systems than those shown in Figure 1. The functionality of the various modules and systems described herein may be implemented by modules and / or systems other than the modules and / or systems described herein.

[0014] In FIG. 1 and other figures, like reference numbers are used to identify like elements. Letters following a reference number, such as "130a," indicate that the text is referring specifically to the element with that particular reference number. A reference number without a following letter, such as "130" in the text, refers to any or all of the elements in the figure with that reference number (e.g., "130" in the text refers to reference numbers "130" and / or "130" in the figures).

[0015] The knowledge-based AI system 150 allows an expert to configure rules for making predictions related to the system. The knowledge-based AI system 150 further generates models, e.g., machine learning models, for making predictions. The knowledge-based AI system 150 combines the results of the rule-based system and the machine learning-based system to make predictions. Further details of the knowledge-based AI system 150 are shown and described in relation to FIG. 2.

[0016] A device can be any physical device, for example a device connected to other devices and systems via the Internet of Things (IoT). The IoT is a network of physical equipment, vehicles, appliances, and other objects equipped with electronics, software, sensors, actuators, and connectivity that allows these objects to connect and exchange data. A device can be a sensor that transmits a sequence of sensed data over time.

[0017] The set of data received from the device may represent data generated by the device, e.g., sensor data, or data obtained by further processing the data generated by the device. Further processing of the data generated by the device may include scaling the data, applying a function to the data, or determining a moving aggregate value based on multiple values ​​generated by the device, e.g., a moving average.

[0018] In an embodiment, device 130 is a client device used by a user to interact with knowledge-based AI system 150. The user of device 130 includes an expert who configures knowledge-based AI system 150. In an embodiment, device 130 executes an application 135 that allows the user to interact with knowledge-based AI system 150. For example, application 135 executing on device 130 may be an internet browser that interacts with a web server executing on knowledge-based AI system 150.

[0019] The system and applications illustrated in FIG. 1 may be implemented using a computing device, such as a Microsoft TM Windows TM Compatible Operating Systems (OS): Apple TM The computing devices may be conventional computer systems running OS X and / or Linux distributions, and may also be client devices with computing capabilities, such as personal digital assistants (PDAs), mobile phones, video game systems, and the like.

[0020] Interaction between device 130 and knowledge-based AI system 150 typically occurs over network 110, e.g., the Internet. In one aspect, the network uses standard communication technologies and / or protocols. In another embodiment, the various entities that interact with each other, e.g., knowledge-based AI system 150 and device 130, may use custom and / or proprietary data communication technologies instead of or in addition to those described above. In some embodiments, the network may also include links to other networks, such as the Internet.

[0021] System Architecture FIG. 2 illustrates a system architecture of a knowledge first system according to an embodiment. The knowledge first system 120 includes a knowledge model 210, a generalized model 220, an ensemble oracle 230, a data synthesizer 240, and a knowledge modeler 250. In other embodiments, the knowledge first system 120 may include more or fewer modules than those illustrated in FIG. 2. Moreover, certain functionality may be implemented by modules other than those described herein. In some embodiments, the various components illustrated in FIG. 2 may be executed by different computer systems. For example, the ensemble oracle 230 may be executed by one or more processors different from the processors that execute the knowledge model 210 and the generalized model 220. Furthermore, the various models of the knowledge first system 120 may be executed using a parallel or distributed architecture for faster execution.

[0022] Knowledge model 210 stores rules based on domain experts. In an embodiment, knowledge model 210 is a rule-based system. The rules may be provided by a domain expert. The rules may incorporate expert-specified thresholds that may be used to predict values ​​or take actions. For example, if a certain input is above a predetermined threshold, a certain action should be taken.

[0023] The generalized model 220 is a trained machine learning based model that makes predictions based on input data. The generalized model 220 may be trained incrementally as new training data becomes available. Thus, the generalized model 220 evolves. For example, the generalized model 220 may be initialized with parameters obtained from a machine learning model trained with a small training data set. Periodically, the generalized model 220 is trained with a larger and better training data set. Thus, the parameters of the generalized model 220 are updated with the better trained model.

[0024] Each of the knowledge model 210 and the generalized model 220 makes a prediction and also outputs a measure of accuracy (or confidence score) related to the predicted output. The accuracy measure of each model is used to determine how to determine the final output based on the output of each of the models, i.e., the knowledge model 210 and the generalized model 220. The accuracy of the generalized model 220 is determined in a model evaluation stage and may be provided with the model, for example, as a function (or instruction set) that calculates the model accuracy. In an embodiment, the knowledge model 210 uses Boolean rules, such as, for example, rules specified as if-then-else statements that compare input data to a threshold to determine an outcome. In another embodiment, the knowledge model 210 uses fuzzy logic with multi-valued variables (compared to Boolean variables that can take only two values). For example, the knowledge model 210 may receive some data and determine statistics that describe the data to generate fuzzy logic.

[0025] The ensemble oracle 230 decides whether to use the prediction of the generalized model 220 or the prediction based on the knowledge model 210. Thus, if the ensemble oracle 230 determines that the prediction of the generalized model 220 is low in accuracy (accuracy below a threshold or confidence score below a threshold), the ensemble oracle 230 uses the prediction of the knowledge model 210. If the ensemble oracle 230 determines that the prediction of the generalized model 220 is accurate (has an accuracy above a threshold or has a confidence score above a threshold), the ensemble oracle 230 uses the prediction of the generalized model 220.

[0026] In an embodiment, the ensemble oracle 230 combines the results of the generalized model 220 and the knowledge model 210 to determine the result. For example, if the output of each of the knowledge model 210 and the generalized model 220 is Boolean, the ensemble oracle 230 performs an AND operation on the outputs of the knowledge model 210 and the generalized model 220 and returns the result of the AND operation as the overall prediction. In an embodiment, the ensemble oracle 230 determines the final result by taking a weighted aggregate of the outputs of the knowledge model 210 and the generalized model 220. The weight assigned to each output may be determined based on a measure of the accuracy of the corresponding model executed to determine the output.

[0027] In an embodiment, the ensemble oracle 230 compares the accuracy of the knowledge models 210 and the generalized models 220 and selects the output of the model with the higher accuracy. In an embodiment, the ensemble oracle 230 is itself a machine learning based model.

[0028] The results of the ensemble oracle 230 are used by the production system for operation. The results may also be stored (e.g., logged) and later used to evaluate models, such as the knowledge model 210 and the generalization model 220. For example, the execution results may be provided by the system to an expert user. The expert user may modify the rules used for future executions or the thresholds used by the rules based on the results of past executions. Thus, the system receives modified rules after the execution results are presented.

[0029] The knowledge model 210 is also used to generate training data, such as, for example, labeling the data used to train the generalized model 220. However, the knowledge model 210 is also used at run time to make predictions when the results of the generalized model are determined to have low accuracy.

[0030] The data synthesizer 240 includes models used to automatically generate data related to a system, for example, an industrial system. The data synthesizer 240 may include a mathematical model that may be provided by an expert. The data synthesizer 240 may include a representation of noise that may be added to the data generated using the mathematical model to determine realistic data that may be used as an initial training data set. The training data set generated by the data synthesizer 240 is used to train the generalized model 220. The models that the data synthesizer 240 uses to generate may be domain specific. However, the data synthesizer 240 may generate data using generic techniques such as Monte Carlo methods.

[0031] In an embodiment, each of the knowledge model 210 and the generalized model 220 may be configured to perform pre-processing of the input data. In an embodiment, the output of each of the knowledge model 210 and the generalized model 220 is of the same format, structure, and type such that the ensemble oracle 230 can combine the two outputs to generate a final output. Although the same raw data is provided as input to both the knowledge model 210 and the generalized model 220, the pre-processing of the two models may differ.

[0032] Knowledge modeler 250 allows experts to configure knowledge model 210. In an embodiment, knowledge modeler 250 configures a user interface and sends it for presentation to an expert user. The expert user can use the user interface to perform operations such as setting thresholds, creating polygons or shapes to create boundaries that mark subsets of data that are relevant to particular semantics, labeling data, and the like.

[0033] Overall Processing Figure 3 illustrates an overall process for making predictions according to an embodiment of the present invention. The steps illustrated in the process may be performed in an order different than that shown in Figure 3. Additionally, the steps are shown as being performed by a system, e.g., knowledge-based AI system 150, and may be performed by suitable modules as shown in and described in connection with the description of Figure 2.

[0034] The system receives 310 input data that needs to be processed to make a prediction. The input data can be sensor data, event data generated by the system, user data, or other types of data that may be provided as input to a model to make a prediction. The system runs 320 the knowledge model 210 with the input data to generate an output, e.g., O1. The system runs 330 the generalization model 220 with the input data to generate another output, e.g., O2. The system determines 340 the accuracy of each of the knowledge model 210 and the generalization model 220. The system determines 350 a final prediction, e.g., O3, based on a combination of the output O2 of the knowledge model 210 and the output O2 of the generalization model 220. The system stores the final prediction O3 and uses it to take further downstream actions. Additionally, the generalization model 220 may be referred to herein as a machine learning based model.

[0035] Artificial Intelligence-Based Systems FIG. 4 shows a development system for use in building an AI system according to an embodiment. The development system is based on a special structure for comprehensive AI systems, i.e. systems that run from development to operation, and is configured with multiple microservices (apps) working together to meet the requirements of the system. A notebook is sufficient for one model, not the entire system. The development system provides the necessary tools to take advantage of individual development streams. For example, a back-end engineer can work on creating a batch inference app even before the model is created, since certain features are guaranteed for all models, whether ML or not. This development system allows multiple people to work on separate development streams simultaneously while maintaining the integrity of the system.

[0036] 5 is a diagram illustrating the overall architecture of a knowledge-based AI system according to an embodiment. The diagram illustrates the interaction between domain experts and various components of a knowledge-based AI system 150 for making predictions.

[0037] 6 is a diagram illustrating the overall process of making predictions using a knowledge-based AI system according to an embodiment. FIG. 6 shows the flow of information through various components of a knowledge-based AI system 150.

[0038] 7 is a diagram illustrating the use of various tools for use by a knowledge-based AI system 150 according to an embodiment. For example, tools such as a knowledge modeler and a machine learning modeler may be used.

[0039] The Knowledge First System 120 can be used in a variety of applications, such as in industrial systems. One example of an application in which the Knowledge First System 120 can be used is predictive maintenance and failure prediction of equipment.

[0040] 8-11 are diagrams illustrating the use of a knowledge-based AI system for an application according to various embodiments.

[0041] The diagram shows an application of the K1st Oracle architecture. This is a generalized application for predictive maintenance, where data first goes through an unsupervised anomaly detection process and then through a k-Oracle. An Oracle is a node where the user provides a knowledge model (teacher) and the system creates a generalized ML model (student) and a default ensemble (which can be customized as well). The teacher is essentially a collection of rules set by a domain expert that dictates what type of fault is associated with a particular pattern in the data. For example, an expert could say, "If the outlet temperature is 40°C higher than the inlet temperature, then there is a coolant leak," and the teacher model would have the rule, "If anomaly & data["outlet_temp"] - data["inlet_temp"] > 40: return "coolant_leak"." In the training process, all data goes through the teacher model, which creates labels that are used to train the student model (in this particular case, the student model uses a naive Bayes classifier as its base, but other implementations use deep neural network models). The advantage of this is that the ML model is more flexible and performs better in edge cases where a hard-lined teacher model would be inaccurate. Finally, the outputs of both models are passed to an ensemble model, which decides the priority of both predictions. In the simplest implementation, the ensemble could simply combine the two inputs (e.g., if the student and teacher output Boolean classifications, an AND or OR gate could be enough), but it could also receive evaluation metrics from the two models and decide based on that whose output to trust, or if the output is a number, it could use an accuracy that weights the outputs and takes the average. All these options are use case specific.At the highest level, with enough labeled training data, the ensemble is implemented as an ML model that can learn on its own how to best leverage the predictions of both models to generate decisions. Most companies with small data sets will start with a logical ensemble, but over time, the system will label the data as usage and occasionally expert evaluation and feedback will be used to edit and correct the dataset, eventually becoming large enough to support training an ML ensemble. The key to the architecture is the k-oracle, which can be implemented and used in a variety of ways. It is an extensible architecture that can fit many use cases, and it serves as a simple way to integrate domain expertise into AI and leverage it to overcome the hurdle of having little to no training data or labels. Of note, the system can be trained even with no data at all. In that situation, the ML model will give effectively random outputs, and the ensemble will only use the teacher output until enough data is available to train the student model.

[0042] Collaboratively augmented language models for developing AI solutions The domain-specific knowledge that needs to be incorporated into the model is often not held by a specific individual, but is distributed among several users. Moreover, the users may not have the complete knowledge up front. The system builds a library or libraries that store the knowledge from different users. When the user builds the model, the knowledge is extracted from the library.

[0043] 12 illustrates the architecture of a system for developing AI solutions according to an embodiment. The system is composed of a knowledge capture module 1210, a knowledge management module 1220, a digital subject matter expert (dSME) module 1230, and a knowledge translation (K-translator) module 1240. Other embodiments may have more or fewer modules than those illustrated in FIG. 12.

[0044] The knowledge capture module 1210 is responsible for extracting expert knowledge (or domain knowledge) from subject matter experts in a conversational manner. The module may use natural language processing (NLP) techniques to understand the knowledge provided by the subject matter experts and work with the experts to deliver the knowledge in a complete and accurate form. A user may upload documents, e.g. PDF documents, text files, doc files, etc., that store domain-specific knowledge to the library. The module tries to understand the domain of the experts by accessing existing knowledge in the catalog and (A) extract more complete information and (B) actively collaborate with the experts to make the knowledge complete and / or correct. According to an embodiment, the knowledge capture module 1210 is a chatbot trained to have a conversation with the domain experts and ask questions to extract information from the domain experts. According to an embodiment, the extracted knowledge is stored in a database, e.g., a GPT index or a Langchain. TMThe domain knowledge is stored in a vector database, such as a vector database. The vector database allows for extracting relevant parts of the stored knowledge to answer domain-specific questions based on the extracted domain knowledge. According to an embodiment, the knowledge capture module 1210 executes a conversation with the user and builds a conversation summary, for example by providing a conversation to a large language model with one or more prompts, requesting the large language model (LLM) to summarize the conversation with the domain expert, and using the LLM's response as the conversation summary. Thus, the knowledge capture module 1210 allows the system to use the LLM to convert the natural language-based domain-specific knowledge obtained from one or more users and convert the knowledge into structured data that is stored in a library, such as a vector database. The system uses the knowledge stored in the library to build a domain-specific model (e.g., a domain-specific LLMS).

[0045] A large-scale language model may also be referred to herein as a machine learning-based language model. In one embodiment, a large-scale language model (LLM) is trained on a large corpus of training data for processing natural language requests. The LLM may be trained on a large amount of text data that often includes billions of words or text units. The LLM may be trained on a large amount of data from various data sources. For example, the data sources include websites, articles, web posts, etc. The large amount of training data from various data sources allows the LLM to generate outputs for many tasks. The LLM may have a significant number of parameters in a deep neural network (e.g., a Transformer architecture), such as, for example, at least 1 billion, at least 15 billion, at least 135 billion, at least 175 billion, at least 500 billion, at least 1 trillion, at least 1.5 trillion parameters.

[0046] In one example, the LLM may be trained and deployed or hosted on a cloud infrastructure service. In one embodiment, the LLM has a generative pre-training (GPT) architecture that includes a set of decoders, each performing one or more operations on input data to the respective decoder. In another embodiment, the LLM has a transformer architecture that includes a set of encoders coupled to a set of decoders. While an LLM with a transformer-based architecture is described as the primary embodiment, it is understood that in other embodiments, the language model can be configured as any other suitable architecture, including, but not limited to, a long short-term memory (LSTM) network, a Markov network, BART, a generative paradoxical network (GAN), a diffusion model (e.g., Diffusion-LM), and the like.

[0047] The knowledge management module 1220 is responsible for cataloging and summarizing the extracted knowledge. The module may use NLP techniques to summarize the knowledge and store it in a knowledge library that is searchable and accessible to users and can capture knowledge not only through conversations with the knowledge capture module 1210 but also in other formats such as word documents and PDFs, serving as a complete knowledge repository for users and organizations.

[0048] The digital subject matter expert module 1230 is responsible for taking the cataloged knowledge and using it to guide data scientists in building AI models. The digital subject matter expert module 1230 may use the knowledge library to provide answers to questions asked by data scientists and may suggest relevant knowledge that may be useful in building AI models. This module (or may use separate knowledge libraries based on multiple modules, topics or other criteria) serves as a single touchpoint between the available knowledge catalog and data scientists building models or other users who want to know specific information from the catalog. Additionally, the digital subject matter expert module 1230 may be referred to herein as a domain-specific model.

[0049] The knowledge translation module 1240 is responsible for translating the extracted knowledge into a structured format, or domain-specific language (DSL), that can be used to build a simplified compilation of the knowledge and an AI model based on that knowledge (and optionally, the corresponding dataset).

[0050] The system offers several advantages compared to existing systems and methods for AI development: The use of a conversational style for capturing expert knowledge allows for capturing expert knowledge in a more natural and user-friendly flow. Cataloging and summarizing the extracted knowledge makes it easier to manage and query the knowledge. Using dSME as a guide for data scientists in building AI models provides a more natural and user-friendly interface for acquiring the knowledge required for model building.

[0051] FIG. 13 illustrates interactions between various components of a system according to an embodiment. The system stores domain-specific LLMs 1315 in a knowledge base 1310. For example, a domain-specific LLM 1315 may be trained with information about manufacturing, another domain-specific LLM 1315 may be trained with information about cold chain, and another domain-specific LLM 1315 may be trained with information about supply chain. The knowledge bases may be provided to tenants of the system, and the tenants may provide tenant-specific knowledge bases 1325. Thus, the system may include tenant-specific LLMs and have tenant-specific domain-specific knowledge. Each domain-specific LLM acts as a digital subject matter expert (dSME).

[0052] The user interacts with the assistant chatbot 1320 (called assistant chatbot or AItomatic assistant chatbot), which interacts with various components. The user may ask the assistant chatbot 1320 a natural language question that provides information about a problem (e.g., of an equipment) requesting the assistant chatbot 1320 to diagnose the problem. The assistant chatbot 1320 answers the question if the assistant chatbot 1320 has the correct information / knowledge, otherwise the assistant chatbot 1320 determines which tool (or component) to interact with to get the answer. The assistant chatbot 1320 interacts with one or more tools to solve the user's problem. The assistant chatbot 1320 may split a request into multiple requests for different tools, interact with them to gather information, and combine the information to solve the problem. According to an embodiment, the assistant chatbot 1320 interacts with the problem solving LLM 1330, which breaks down the user's problem into smaller problems or steps.

[0053] According to an embodiment, the system receives sensor data. The system interacts with a domain-specific LLM to identify the specific information needed to solve a particular problem. The system builds a model to solve that particular problem. The system then uses the model to solve the problem for new data received. If the system encounters a problem it cannot solve, it sends a message to an expert for help.

[0054] Figure 14 illustrates four knowledge-first (K1st) architectures, each solving a different type of data and prediction problem.

[0055] K-Oracle and K-Collaborator are similar except for the fact that K-Oracle is used when you don't have data labels (or don't have good data labels) and instead use a knowledge model as a teacher for an ML "student" model, whereas K-Collaborator is used when you have ground truth labels for your data but want the benefit of knowledge.

[0056] Both of these architectures are used in small data scenarios when the system does not have enough data or enough examples of important predictive events, and human knowledge is leveraged to capture events that data-only ML would normally miss or fail to learn. These two architectures can utilize rule type knowledge (e.g., If X then Y) to accomplish classification tasks, or rule (e.g., If X then Y changes by Z) and equation (e.g., Y = aX + b) type knowledge to accomplish regression or prediction tasks.

[0057] Furthermore, while a K-Oracle consists of a single knowledge "teacher" model and one or many ML "student" models, a K-Collaborator can contain any number of knowledge and ML models. In both cases, the ensemble can be an algorithm (knowledge-based or ML) that determines the best combination, comparison, and selection among the upstream model outputs.

[0058] The K-SWE architecture is best suited for cases where there is a moderate or large amount of data, but a large variance in the output variable (e.g., when predicting the number of customers or total sales at a convenience store, there is a large variance in the number of customers or total sales by store and time). This architecture takes advantage of the fact that machine learning strives to learn the inherent trends and relationships in the data space, but can fail because (A) there is insufficient data to fully learn all the necessary relationships, (B) the trends are not well represented in the dataset, or (C) the dataset does not have the best features to mathematically understand the trends. However, experts already understand many of the major trends that separate the data space. Therefore, by using their knowledge of trend types (e.g., A and B are more similar than A and C, or Y is more stable for each group within feature X), they can remove the major trends in the data and partition the data into discrete subspaces such that ML can better learn the remaining trends with less data. For example, suppose I am building a model to predict the number of people visiting a certain park, I, as an expert, know that there are clear differences in park visitor activity and weather across different seasons. That is, instead of fitting one model across all months and cities to predict the number of people in the park, it is possible to fit one model in summer, another model in winter, one model for rainy days, one model for sunny days, and one model for sunny days in summer to predict the number of people going to the park. Inference runs only the relevant models, and for sunny summer days, three models are run: a summer model, a sunny model, and a sunny summer model. Furthermore, the system inherently scales to predict situations that were not in the original dataset. Suppose there were no sunny winter days in the initial dataset, but there were sunny winter days during operation. Then two models are run (a winter model and a sunny model) and the outputs of these two are ensembled.

[0059] Furthermore, combining the model with other K1st models is possible since a "subspace ML model" can actually be any type of predictive model. In this way, it is possible to have a K-SWE model where each subspace model is a K-oracle, eliminating the need for data labels.

[0060] The K-CP (Conditional Probability) architecture combines ML and knowledge, but uses knowledge to directly influence the probability space of the ML model. ML models inherently compute the conditional probability of a potential output as the probability of output Y given data X. We can similarly create a conditional probability model that also computes the probability of output Y given data X, and use knowledge to combine this with the ML probability before the final stage of ML processing (thresholding, softmax, weighted average, etc.). This architecture is particularly useful when knowledge is applied to information that is invisible to the ML model. A perfect example is the case of computational vision, where the ML model only looks at an image and makes some prediction. The ML model sees the image, but does not see (and therefore act on) the metadata associated with the image (e.g. the time, place, external conditions the image was taken, etc.). In this case, the ML model operates on the image to generate the probability of output Y given data X, and the knowledge model (a conditional probability model) operates on the metadata to generate the probability of output Y given metadata Z. Now, both probabilities can be combined to give a more informed (and therefore more accurate) result.

[0061] The method can also be used when both the ML and knowledge models are looking at the same data, since a human can review the model's output and directly inject knowledge to correct what the ML is explicitly failing to learn (usually due to insufficient data for the ML to learn from).Alternatively, the architecture just described can be used to intentionally bias an AI system that uses knowledge to prevent adverse outcomes, such as intentionally reducing false negatives in favor of false positives in mission-critical systems (because it is safer to react to failures that do not occur than to miss failures that do occur).

[0062] Applications of AI systems The disclosed system, referred to herein as the CALM system, has been applied or is designed to be applied to a variety of applications, such as:

[0063] The system can be applied to predictive maintenance of industrial equipment. By integrating the CALM system with various sensors and data sources, it will be possible to predict equipment failures in advance, reducing downtime and maintenance costs of industrial equipment.

[0064] The system is applicable to autonomous vehicle control. The CALM system could be used to integrate multiple domain-specific models and human operators to enable autonomous control of cars. What has been described could improve safety and efficiency in a variety of environments, such as shipping ports and large-scale mining operations.

[0065] The system is applicable to energy optimization in smart buildings. By integrating the CALM system with smart building technologies and data sources, the system can be used to optimize a building's energy usage and reduce waste. This could lead to significant cost savings and environmental benefits.

[0066] The system is applicable to medical diagnosis and treatment. The CALM system could be used to integrate multiple domain-specific models and human operators to assist in medical diagnosis and treatment planning. For example, the system could be used to combine data from medical image scans, clinical tests, and patient medical history to suggest more accurate diagnoses and treatments.

[0067] The system is applicable to environmental monitoring and control. The CALM system could be used to integrate data from various environmental sensors and models to more effectively monitor and manage natural resources such as water and air quality. This could bring great benefits to public health and environmental conservation efforts.

[0068] 15 shows a flow chart illustrating a process for answering domain-specific natural language requests according to an embodiment. The steps may be performed by modules of a system, such as knowledge-based AI system 150. The steps may be performed in a different order than shown herein.

[0069] The system receives 1510 a natural language request. The natural language request represents a domain-specific problem that a user is interested in solving. The request may be received by a chatbot 1320 configured to receive and process the natural language request, perform actions based on the natural language request, and generate a response based on the natural language request. For example, the natural language request may be specific to an industry and to request a solution to a problem encountered with equipment used in that industry. The natural language request may specify details of the equipment and a symptom of the problem, e.g., the equipment is a particular type of refrigeration equipment (specified using a vendor name and model number) and the problem is that the refrigeration equipment is not cooling below a particular point. The natural language request may further specify details of the system environment of the equipment, e.g., the type of facility in which the refrigeration equipment is installed, the ambient temperature, the size of the area cooled by the refrigeration equipment, etc.

[0070] The system determines 1520 whether a domain-specific model (also referred to as a dSME) accessible to the system can solve the domain-specific problem specified by the natural language request. For example, the system determines whether a particular domain-specific knowledge is based on the knowledge required to solve the domain-specific problem specified by the natural language request. If a domain-specific model based on the knowledge of the domain of the domain-specific problem exists, the system selects 1530 the domain-specific model from the multiple domain-specific models accessible to the system. The system executes 1540 the domain-specific model to generate a response. For example, the LLM may return a response in a format requested in the prompt, such as a JSON (JavaScript Object Notation) object. The system determines 1550 a result based on the response, for example, by extracting attributes of the JSON object that represent the result. The system provides 1560 the result to the user, for example, by sending the result to the user's client device.

[0071] If the system determines 1520 that none of the available domain-specific models accessible to the system can solve the domain-specific problem specified by the natural language request, e.g., none of the available domain-specific models store or have been trained using data for that particular domain, then the system uses a software tool to solve the problem. Thus, the system selects 1535 a software tool from a plurality of software tools accessible to the system, e.g., a search engine. The system may use the LLM to select the software tool, e.g., by providing a prompt to the LLM describing the available software tools and the natural language request, and requesting the LLM to identify the best software tool to solve the domain-specific problem.

[0072] The system executes the selected software tool and generates a response 1545. For example, the system may extract one or more search keywords from the natural language request and generate a search request to a search engine consisting of the search keywords. The search engine may return a set of search results that match the search request. The system determines results based on the response 1555, for example, by extracting search results from the set of search results. The system provides the results to the user 1560, for example, by sending the results to the user's client device.

[0073] According to an embodiment, the system uses the LLM to determine the best search results from a set of search results that match the search request. The system generates a prompt that includes one or more search results, a natural language request, and a request to determine whether the search results contain information to answer a domain-specific question specified by the natural language request. The system provides the prompt as an input to the LLM. The system receives a response obtained by executing the LLM. The system selects a result from the set of search results based on the response obtained by executing the large-scale language model.

[0074] According to an embodiment, the system may access multiple domain-specific models, each configured to solve a problem in the domain. The system may save descriptions of the domain-specific models and compare the natural language request to the domain-specific models. According to an embodiment, the system stores the descriptions of the domain-specific models in a vector database. For example, the system may generate a vector representation of each description of the domain-specific models and store the vector representations in the vector database. The vector representation of the natural language text may be generated, for example, by providing the natural language text as an input to a neural network, such as a multi-layer perceptron, trained to encode and decode the input natural language text. The vector representation may be the output of a hidden layer of the neural network that processes the natural language text.

[0075] The system may further generate a vector representation of the natural language request and compare the vector representation of the natural language request with the vector representation of the description of the domain-specific model to determine which domain-specific model is closest to the natural language request. The comparison may be based on a vector distance metric, e.g., a cosine similarity measure. If the distance of all of the vector representations of the description of the domain-specific model from the vector representation of the natural language request exceeds a predetermined threshold, the system may determine 1520 that no domain-specific model exists that is capable of solving the problem described in the natural language request.

[0076] According to another embodiment, the system uses the large-scale language model to determine 1520 an appropriate domain-specific model. The system generates a prompt including a description of each of the multiple domain-specific models, a natural language request, and a request to determine whether any of the multiple domain-specific models can solve the domain-specific problem specified in the natural language request. The system provides the prompt to the LLM for execution. The LLM may be executed by an external system, such as a web service, or may be stored and executed locally. If the LLM is executed by the external system, the external system executes the LLM, such as by sending an execution request of an API (Application Programming Interface) of the external system over a network. If the LLM is executed locally within the system, the system may call a function that executes the LLM. The system receives a response resulting from the execution of the large-scale language model. The system extracts an indication from the response of whether any of the multiple domain-specific models is capable of solving the domain-specific problem specified in the natural language request.

[0077] According to an embodiment, the system may indicate to the prompt that if any of the multiple domain-specific models are not capable of solving the domain-specific problem, the LLM should select a domain-specific model for solving the domain-specific problem specified by the natural language request. Thus, the LLM selects one of the domain-specific models if the domain-specific model has the necessary knowledge to solve the domain-specific problem specified by the natural language request. The system extracts information from the response that identifies the domain-specific model that can solve the domain-specific problem specified in the natural language request.

[0078] According to an embodiment, the domain-specific model includes (1) a machine learning-based model trained to make a prediction based on specific input data, (2) a knowledge model, where the knowledge model is a rule-based model, where each rule makes a prediction based on one or more characteristics of the input data, and (3) an ensemble model configured to combine results of the knowledge model and the machine learning-based model. The ensemble model combines results of the knowledge model and the machine learning-based model.

[0079] A knowledge model may store rules based on domain experts. In an embodiment, the knowledge model is a rule-based system. The rules may be provided by a domain expert. The rules may incorporate expert-specified thresholds that may be used to predict values ​​or take actions. For example, if a certain input is above a predetermined threshold, a certain action should be taken.

[0080] A machine learning based model of a domain-specific model is a trained machine learning based model that makes predictions based on input data. A machine learning based model may be incrementally trained as new training data becomes available. Accordingly, the machine learning based model evolves. For example, a machine learning based model may be initialized with parameters obtained from a machine learning model trained with a small training data set. Periodically, the machine learning based model is trained with a larger and better training data set. Thus, the parameters of the machine learning based model are updated using the better trained model. Knowledge models are also used to generate training data, e.g., labeling data used to train the machine learning based model.

[0081] According to an embodiment, the ensemble model combines the results of the knowledge model and the machine learning based model based on the accuracy measure of the knowledge model and the accuracy measure or accuracy of the machine learning based model. The accuracy measure of each model may be one of the outputs predicted by the model. For example, the machine learning based model may be trained to predict the accuracy measure along with the outcome. Similarly, a rule based system may include rules for predicting the accuracy of an outcome.

[0082] According to an embodiment, if the ensemble model determines that the prediction of the machine learning-based model of the domain-specific model is inaccurate (the accuracy is below a threshold or the confidence score is below a threshold), the ensemble model uses the prediction of the knowledge model. If the ensemble model determines that the prediction of the machine learning-based model of the domain-specific model is accurate (the accuracy is above a threshold or the confidence score is above a threshold), the ensemble model uses the prediction of the machine learning-based model.

[0083] According to an embodiment, the knowledge model uses Boolean rules, such as rules specified as if-then-else statements that compare input data to thresholds to determine an outcome. In another embodiment, the knowledge model uses fuzzy logic with multi-valued variables (as compared to Boolean variables that can take only two values). For example, the knowledge model may receive some data and determine statistics that describe the data to generate fuzzy logic.

[0084] According to an embodiment, the ensemble model is configured to select an output of the machine learning based model as the final output if the accuracy measure of the machine learning based model is higher than the accuracy measure of the knowledge model, and to select an output of the knowledge model as the final output if the accuracy measure of the knowledge model is higher than the accuracy measure of the machine learning based model. According to an embodiment, the ensemble model generates a final output that is a weighted aggregate of the output of the machine learning based model and the output of the knowledge model, where the weight of each output is determined based on the accuracy measure of the corresponding model.

[0085] Computer Architecture 16 is a high-level block diagram illustrating an exemplary system according to an embodiment. The computer 1600 includes at least one processor 1602 connected to a chipset 1604. The chipset 1604 includes a memory controller hub 1620, and an input / output (I / O) controller hub 1622. Memory 1606 and a graphics adapter 1612 are connected to the memory controller hub 1620, and a display 1618 is connected to the graphics adapter 1612. A storage device 1608, a keyboard 1610, a pointing device 1614, and a network adapter 1616 are connected to the I / O controller hub 1622. Other embodiments of the computer 1600 have different architectures.

[0086] The storage device 1608 is a non-transitory computer-readable recording medium, such as, for example, a hard drive, a compact disk read-only memory (CD-ROM), a DVD, or a solid-state memory device. The memory 1606 holds instructions and data used by the processor 1602. The pointing device 1614 is a mouse, trackball, or other type of pointing device and is used in combination with the keyboard 1610 to input data into the computer system 1600. The graphics adapter 1612 displays images and other information on the display 1618. The network adapter 1616 connects the computer system 1600 to one or more computer networks.

[0087] The computer 1600 is adapted to execute computer program modules for providing the functionality described herein. In this specification, the term "module" refers to computer program logic used to provide a specified function. Thus, a module can be implemented in hardware, firmware, and / or software. In one embodiment, the program module is stored in the storage device 1608, loaded into the memory 1606, and executed by the processor 1602. The type of computer 1600 used can vary depending on the embodiment and requirements. For example, the computer may not have a display, keyboard, and / or other devices shown in FIG. 16.

[0088] Additional Considerations It should be understood that the figures and descriptions of the present invention are simplified to illustrate elements relevant for a clear understanding of the present invention, while excluding, for clarity, many other elements found in a typical distributed system. Those skilled in the art may recognize that other elements and / or steps are desirable and / or necessary in implementing the embodiments. However, because such elements and steps are well known in the art and do not facilitate a better understanding of the embodiments, a discussion of such elements and steps is not provided herein. The disclosure herein is directed to all such variations and modifications to such elements and methods known to those skilled in the art.

[0089] Some portions of the above description describe aspects in terms of algorithmic processes or operations. The presently described algorithmic descriptions and representations are commonly used by data processing engineers to effectively convey the substance of their work to others skilled in the art. The presently described operations, while described functionally, computationally, or logically, will be understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Moreover, it has also proven convenient at times, without loss of generality, to refer to the presently described arrangements of operations as modules. The described operations and associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0090] Any reference to "one embodiment" or "an embodiment" as used herein means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

[0091] Some embodiments may be described using the terms "coupled" and "coupled" along with their derivatives. It should be understood that these terms are not intended as synonyms for each other. For example, some embodiments may be described using the term "connected" to indicate that two or more elements are in direct physical or electrical contact with each other. In another example, some embodiments may be described using the term "coupled" to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other. The embodiments are not limited in this context.

[0092] As used herein, the terms "comprises," "comprising," "including," "including," "having," "having," or any other variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to only those elements just mentioned, but may include other elements not expressly listed or inherent to the process, method, article, or apparatus. Furthermore, unless expressly stated to the contrary, "or" refers to an inclusive or and not an exclusive or. For example, condition A or B satisfies any one of the following: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), or both A and B are true (or exist).

[0093] Furthermore, the use of "a" or "an" is employed to describe elements and components of embodiments herein. This is merely for convenience and to give a general sense of the invention. The description just given should be read to include one or at least one, unless it is clear that something else is meant, and furthermore the singular also includes the plural.

[0094] Upon reading this disclosure, those skilled in the art will still appreciate additional alternative structural and functional designs for systems and processes for displaying charts using distortion regions through the principles disclosed herein. Thus, although specific aspects and applications have been illustrated and described, it is to be understood that the subject matter described is not limited to the disclosed aspects to the precise structures and components disclosed herein. Various modifications, changes and variations may be made in the arrangement, operation and details of the methods and apparatus disclosed herein without departing from the spirit and scope as defined in the appended claims, as will be apparent to those skilled in the art.

Claims

1. A computer-readable non-transitory memory storing instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform method steps for solving a domain-specific problem specified using a natural language request, the method steps comprising: receiving, by the chatbot, a natural language request for answering a question from one of a plurality of domains, the natural language request including a domain-specific problem; determining whether any of a plurality of domain-specific models is configured to answer the domain-specific problem, each of the plurality of domain-specific models storing a domain-specific knowledge base; in response to determining that at least one of the plurality of domain-specific models is configured to answer the domain-specific problem specified in the natural language request, selecting a domain-specific model from the plurality of domain-specific models for answering the domain-specific problem; executing the domain-specific model to answer the domain-specific problem; in response to determining that none of the plurality of domain-specific models is capable of answering the domain-specific problem, selecting a software tool from a plurality of software tools to answer the domain-specific problem; sending a request to the software tool to answer the domain specific problem; sending results of execution of said software tool to a client device; 1. A computer readable non-transitory memory comprising:

2. The instructions cause the one or more computer processors to: for each of the plurality of domain-specific models, storing a vector representation of a description of the domain-specific model in a vector database; determining a vector representation of the natural language request; for each of the plurality of domain-specific models, determining a vector distance between the vector representation of the natural language requirement and the vector representation of the description of the domain-specific model; The computer-readable non-transitory memory of claim 1, further comprising the steps of:

3. The instructions for determining whether any of a plurality of domain-specific models is configured to answer the natural language request may further include instructions for the one or more computer processors to: determining, for each of the plurality of domain-specific models, in response to determining that the vector distance between the vector representation of the natural language request and the vector representation of the description of the domain-specific model exceeds a threshold, that none of the domain-specific models is capable of solving the domain-specific problem specified in the natural language request.

3. The computer-readable non-transitory memory of claim 2, further comprising the steps of:

4. The instructions cause the one or more computer processors to: a description of each of the plurality of domain-specific models; and the natural language request; a request to determine whether any of the plurality of domain-specific models is capable of solving the domain-specific problem specified in the natural language request; generating a prompt including: sending the prompts to a large-scale language model for execution; receiving a response obtained by executing the large-scale language model; extracting from the response an indication of whether any of the plurality of domain-specific models is capable of solving the domain-specific problem specified in the natural language request. The computer-readable non-transitory memory of claim 1, further comprising the steps of:

5. The prompt further requests the large scale language model to select a domain-specific model for solving the domain-specific problem if any of the plurality of domain-specific models is capable of solving the domain-specific problem, and the instructions cause the one or more computer processors to: extracting from the response a domain-specific model configured to solve the domain-specific problem specified in the natural language request. The computer-readable non-transitory memory of claim 4, further comprising the steps of:

6. At least the domain-specific model A machine learning-based model that is trained to make predictions based on specific input data; a knowledge model, the knowledge model being a rule-based model, each rule making a prediction based on one or more characteristics of input data; an ensemble model configured to combine results of the knowledge model and the machine learning based model; 2. The computer readable non-transitory memory of claim 1, comprising:

7. 7. The computer-readable non-transitory memory of claim 6, wherein the ensemble model combines results of the knowledge model and results of the machine learning based model based on a measurement of accuracy of the knowledge model and a measurement or accuracy of the machine learning based model.

8. 8. The computer-readable non-transitory memory of claim 7, wherein the ensemble model is configured to select an output of the machine learning based model as a final output if the measure of accuracy of the machine learning based model is higher than the measure of accuracy of the knowledge model, and to select the output of the knowledge model as the final output if the measure of accuracy of the knowledge model is higher than the measure of accuracy of the machine learning based model.

9. 7. The computer-readable non-transitory memory of claim 6, wherein the ensemble model generates a final output that is a weighted aggregate of the outputs of the machine learning based models and the knowledge models, the weight of each output being determined based on a measure of accuracy of the corresponding model.

10. At least one software tool is a search engine, and the instructions cause the one or more computer processors to: extracting one or more search keywords from the natural language request; providing the one or more search keywords to the search engine; receiving a set of search results from the search engine that match the one or more search keywords; extracting a response to the natural language request from a search result selected from the set of search results; 2. The computer-readable non-transitory memory of claim 1, further comprising:

11. The instructions for extracting the response to the natural language request from the search results selected from the set of search results may include directing the one or more computer processors to: generating a prompt including one or more search results, the natural language request, and a request to determine whether the search results contain information to answer the domain-specific question specified by the natural language request; providing the prompt as an input to a large-scale language model; selecting a result from the set of search results based on a response obtained by executing the large-scale language model; 11. The computer readable non-transitory memory of claim 10, further comprising:

12. 1. A computer-implemented method for solving a domain-specific problem specified using a natural language request, comprising: receiving, by a chatbot, a first natural language request for answering a question from one of a plurality of domains, the first natural language request including a first domain-specific problem; determining whether any of a plurality of domain-specific models is configured to answer the first domain-specific problem, each of the plurality of domain-specific models storing a domain-specific knowledge base; in response to determining that at least one of the plurality of domain-specific models is configured to answer the domain-specific problem specified in the natural language request, selecting a domain-specific model from a plurality of domain-specific models for answering the first domain-specific problem; executing the domain-specific model to answer the first domain-specific problem; receiving, by the chatbot, a second natural language request for answering a question from one of a plurality of domains, the second natural language request including a second domain-specific problem; in response to determining that none of the plurality of domain-specific models is capable of answering the second domain-specific problem, selecting a software tool from a plurality of software tools to answer the second domain-specific problem; sending a request to the software tool to answer the second domain-specific problem; sending results of execution of said software tool to a client device; 16. A computer-implemented method comprising:

13. for each of the plurality of domain-specific models, storing a vector representation of a description of the domain-specific model in a vector database; determining a vector representation of the natural language request; for each of the plurality of domain-specific models, determining a vector distance between the vector representation of the natural language requirement and the vector representation of the description of the domain-specific model; 13. The computer-implemented method of claim 12, further comprising:

14. Determining whether any of a plurality of domain-specific models is configured to answer the natural language request includes: determining, for each of the plurality of domain-specific models, in response to determining that the vector distance between the vector representation of the natural language request and the vector representation of the description of the domain-specific model exceeds a threshold, that none of the domain-specific models is capable of solving the domain-specific problem specified in the natural language request.

14. The computer-implemented method of claim 13, comprising:

15. a description of each of the plurality of domain-specific models; and the natural language request; a request to determine whether any of the plurality of domain-specific models is capable of solving the domain-specific problem specified in the natural language request; generating a prompt including: sending the prompts to a large-scale language model for execution; receiving a response obtained by executing the large-scale language model; extracting from the response an indication of whether any of the plurality of domain-specific models is capable of solving the domain-specific problem specified in the natural language request.

13. The computer-implemented method of claim 12, further comprising:

16. At least the domain-specific model A machine learning-based model that is trained to make predictions based on specific input data; a knowledge model, the knowledge model being a rule-based model, each rule making a prediction based on one or more characteristics of input data; an ensemble model configured to combine results of the knowledge model and the machine learning based model; 13. The computer-implemented method of claim 12, comprising:

17. 17. The computer-implemented method of claim 16, wherein the ensemble model is configured to: select an output of the machine learning based model as a final output if a measure of accuracy of the machine learning based model is higher than the measure of accuracy of the knowledge model; and select the output of the knowledge model as the final output if a measure of accuracy of the knowledge model is higher than the measure of accuracy of the machine learning based model.

18. 17. The computer-implemented method of claim 16, wherein the ensemble model generates a final output that is a weighted aggregate of the outputs of the machine learning based models and the knowledge models, where the weight of each output is determined based on a measure of accuracy of the corresponding model.

19. At least one software tool is a search engine, and said computer-implemented method comprises: extracting one or more search keywords from the natural language request; providing the one or more search keywords to the search engine; receiving a set of search results from the search engine that match the one or more search keywords; extracting a response to the natural language request from a search result selected from the set of search results; generating a prompt including one or more search results, the natural language request, and a request to determine whether the search results contain information to answer the domain-specific question specified by the natural language request; providing the prompt as an input to a large-scale language model; and selecting a result from the set of search results based on a response obtained by executing the large-scale language model; Including The computer-implemented method of claim 12 further comprising:

20. 1. A computer system for solving domain-specific problems specified using natural language requests, comprising: one or more computer processors; When executed by the one or more computer processors, it causes the one or more computer processors to: receiving, by the chatbot, a natural language request for answering a question from one of a plurality of domains, the natural language request including a domain-specific problem; determining whether any of a plurality of domain-specific models is configured to answer the domain-specific problem, each of the plurality of domain-specific models storing a domain-specific knowledge base; in response to determining that at least one of the plurality of domain-specific models is configured to answer the domain-specific problem specified in the natural language request, selecting a domain-specific model from the plurality of domain-specific models for answering the domain-specific problem; executing the domain-specific model to answer the domain-specific problem; in response to determining that none of the plurality of domain-specific models is capable of answering the domain-specific problem, selecting a software tool from a plurality of software tools to answer the domain-specific problem; sending a request to the software tool to answer the domain specific problem; sending results of execution of said software tool to a client device; a computer-readable non-transitory memory storing instructions for performing steps including: A computer system comprising:

Images