Domain specific retrieval-augmented generation for industrial applications

Abstract

To provide improved question answering in industrial generative AI.SOLUTION: A system answers natural language questions using a large-scale language model (LLM) and using retrieval-augmented generation. The system stores a set of domain specific documents in a vector database in a manner of being usable by the LLM. The system receives a natural language question. The system retrieves a subset of documents relevant to the natural language question from the vector database. The system determines prior knowledge information required in addition to the subset of documents retrieved from the vector database for answering the natural language question. The system generates a prompt for a machine learning based language model including instructions to the machine learning based language model to refrain from using prior knowledge obtained by the machine learning based language model during training of the machine learning based language model. The system receives a response generated by executing the machine learning based language model based on the prompt. The system performs an action based on the response.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates generally to artificial intelligence and machine learning techniques, and more particularly to domain-specific search expansion generation-based artificial intelligence techniques for industrial applications.

[0002] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit under 35 U.S.C. §119(e) of U.S. Provisional Patent Application No. 63 / 640,155, filed April 29, 2024, which is incorporated by reference in its entirety. [Background technology]

[0003] Artificial intelligence (AI) technologies are useful in several industrial systems. For example, machine learning-based language models are used to generate answers to various problems encountered in various environments. Examples of such machine learning-based language models include large-scale language models (LLMs) such as GPT (generative pretrained transformers). These models are trained using large corpora of text, such as the Internet or book libraries. As a result, such language models are trained to answer a general set of questions. However, these language models lack the domain-specific knowledge necessary to answer questions related to domain-specific problems, such as questions related to specific industrial environments. Such questions require specialized knowledge. Furthermore, these important applications cannot tolerate hallucinations experienced with state-of-the-art language models. For example, large-scale language models may invent facts and use them in answers. However, such invented facts are not real and cannot be used in real industrial environments. As a result, the answers obtained from such language models are often insufficient. Summary of the Invention

[0004] The system answers natural language questions using retrieval-augmented generation. The system stores a set of documents in a vector database. According to an embodiment, the vector database stores domain-specific documents related to a particular domain, for example, an industry domain from an industry such as the semiconductor industry, the oil and natural gas industry, or the manufacturing industry.

[0005] The vector database stores a vector representation of each document. The system receives a natural language question and generates a vector representation of the natural language question. The system retrieves a subset of documents related to the natural language question based on the vector representation of the natural language question. The system determines prior knowledge information needed to add to the subset of documents retrieved from the vector database to answer the natural language question. The system identifies a prior knowledge source system for accessing the prior knowledge information. The system accesses the prior knowledge source system to extract the prior knowledge information.

[0006] The system generates a prompt for the machine learning-based language model that includes (1) a natural language question, (2) a subset of documents retrieved from a vector database, (3) prior knowledge information, and (4) instructions to the machine learning-based language model to not use prior knowledge obtained by the machine learning-based language model during training of the machine learning-based language model. The system provides the prompt to the machine learning-based language model and receives a response generated by running the machine learning-based language model based on the prompt. The system performs an action based on the response.

[0007] Embodiments include a computer-implemented method for performing the steps for answers based on search expansion generation for natural language questions described herein; a non-transitory computer-readable storage medium storing instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform the steps for answers based on search expansion generation for natural language questions described herein; and a computer system including one or more computer processors and a non-transitory computer-readable storage medium storing instructions that, when executed by the one or more computer processors, cause the one or more computer processors to perform the steps for answers based on search expansion generation for natural language questions described herein. [Brief explanation of the drawings]

[0008] The disclosed embodiments have other advantages and features that will become more readily apparent from the detailed description, the appended claims, and the accompanying figures (or drawings), a brief description of which follows. [Figure 1A] FIG. 1A illustrates the overall process performed by a system according to an embodiment. [Figure 1B] FIG. 1B is a diagram illustrating a hierarchical decomposition of a task into subtasks, according to an embodiment. [Figure 1C] FIG. 1C is a diagram illustrating hierarchical task planning performed by a system, according to an embodiment. [Figure 1D] FIG. 1D illustrates the OODA process followed by a system, according to an embodiment. [Figure 2] FIG. 2 is a diagram detailing the process the system follows to answer a query, according to an embodiment. [Figure 3] FIG. 3 illustrates an evaluation framework for determining whether a system can answer a query based on currently available information, according to an embodiment. [Figure 4]FIG. 4 illustrates an exemplary RAG-based system based on LLM that provides improved accuracy of results, according to an embodiment. [Figure 5] FIG. 5 is a flowchart illustrating an overall process for answering domain-specific natural language questions using search expansion generation, according to an embodiment. [Figure 6] FIG. 6 is a high-level block diagram illustrating an exemplary system, according to an embodiment.

[0009] The features and advantages described in the specification are not all-inclusive, and many additional features and advantages will be apparent to those skilled in the art, especially upon consideration of the drawings, specification, and claims. Furthermore, it should be noted that the terminology used herein has been selected primarily for ease of reading and description, and not to limit or restrict the disclosed subject matter. DETAILED DESCRIPTION OF THE INVENTION

[0010] A system according to an embodiment provides improved question answering in industrial generative AI. The system integrates domain-specific model fine-tuning and an iterative inference mechanism into a search and augmentation generation (RAG) workflow. The system achieves improved performance by leveraging better retrievers and generators and using multi-stage inference. The system performs hierarchical task planning, decomposing complex tasks into subtasks, and performing OODA inference, a multi-stage inference loop performed for each task. The system performs iterative inference using the OODA (observe, orient, decide, and act) loop, refining the process through phases of observe, orientation, decision, and action, resulting in answers approaching the quality of human experts.

[0011] (System Overview) The system includes a framework designed to significantly improve the performance of question-answering systems used in industrial environments. By incorporating domain-specific fine-tuning of both the search and generative models, along with the innovative application of an iterative inference mechanism, the system achieves significant improvements in providing accurate and relevant answers. The system utilizes advanced embedding models fine-tuned to capture the nuances of domain-specific terminology, ensuring the search process is highly accurate and tailored to the specific needs of the industry.

[0012] Further enhancing the system's capabilities is the use of domain-adaptive large-scale language models (LLMs) for answer generation. Enhanced by fine-tuning with domain-specific data, these models generate answers that are contextually relevant and conform to the desired domain-specific representational and logical structures. The system ensures that the generated answers meet the high standards expected in professional and industrial contexts and closely mimic the depth of understanding and reasoning provided by human experts.

[0013] The system performs hierarchical task planning by breaking down complex tasks into smaller subtasks and solving them. This can be a recursive process that divides subtasks into even smaller subtasks as needed. Embodiments of the system implement the OODA loop—observe, orient, decide, act—for iterative reasoning. This technique allows for the continuous refinement of answers through successive iterations, increasing the system's ability to handle complex queries with a level of accuracy and relevance approaching that of a human expert. Systematically applying this loop allows the system to dynamically adjust its strategy based on feedback, enabling a sophisticated understanding and handling of complexity in the questions it encounters. This iterative process not only optimizes system performance but also reflects the adaptability and iterative nature of human problem solving, making it well-suited for problem solving in the field of industrial generative AI.

[0014] 1A illustrates the overall process performed by a system according to an embodiment. The system executes the following phases: Observe phase 110, Inspire phase 120, Decide phase 130, and Act phase 140. In the Observe phase 110, the system identifies the problem and determines the scope of available knowledge. In the Inspire phase 120, the system determines what operations can be performed with the available information. In the Decide phase 130, the system determines how to process the information and generates a plan. In the Act phase 140, the system executes the plan and evaluates it to determine whether it worked. Thus, the system implements a reasoning framework based on the OODA loop.

[0015] 1B illustrates a hierarchical decomposition of a task into subtasks, according to an embodiment. Thus, the system divides a complex task into subtasks that are easier to perform, or a complex problem into smaller problems that are easier to answer.

[0016] 1C illustrates hierarchical task planning performed by a system according to an embodiment. The system performs hierarchical task planning to handle multi-step workflows and solve complex problems. The system can reason about complex problems and process them in a logical order. The system uses pattern recognition to perform reasoning.

[0017] 1D illustrates the OODA process followed by a system according to an embodiment, showing the Observe, Orient, Decide, and Act steps performed by the system.

[0018] According to an embodiment, domain-specific knowledge may be available as documents, e.g., documents accessible within an organization. The system encodes the documents into embeddings and stores the document embeddings in a vector database, e.g., a structured index for processing in conjunction with the LLM. Examples of structured indexes include GPT-Index or LlamaIndex. The system receives a query and accesses relevant pieces of domain-specific information from the vector database. The system adds the relevant pieces of domain-specific information to prompts generated for the LLM. In contrast to traditional RAG, this component acts as an A-Augmenter in the RAG by adding to the retrieved relevant chunks.

[0019] The system performs search, expansion, and generation, enhancing each stage compared to traditional approaches. For example, the system enhances the expansion step by allowing experts to add new heuristics and domain-specific rules that are built into the system. Thus, the system adds domain expertise during the expansion phase of the RAG framework. The system may observe a specific situation in an industrial environment, for example, the temperature of some equipment exceeds a threshold. The vector database may not contain a document containing information to resolve the current situation. However, the expert may have the knowledge to resolve the situation by adjusting other parameters, such as the pressure of the equipment. The system allows experts to add rules to the system's knowledge. Such rules provide highly domain-specific solutions to specific situations that may be encountered in an industrial environment.

[0020] 2 details the process a system follows to answer a query, according to an embodiment. The system receives a query 205. The system performs an observation phase 210, in which the system searches available documents, e.g., documents stored in a vector database, to determine whether the available information is relevant to the query 205. According to an embodiment, the system generates an embedding based on the query 205 and performs a nearest neighbor search in the vector database using a similarity metric (e.g., cosine similarity) to identify relevant documents.

[0021] The system performs an orientation phase 220 to determine the information needed to respond to query 205. In this phase, the system determines whether the information available to the system can answer query 205. According to an embodiment, the system uses a multi-question based evaluation framework to determine whether the system can answer the question based on the information currently available to it.

[0022] The system performs a decision phase 230 to determine, based on the available information, whether the system can answer the query 205. If the system determines that the query 205 can be answered based on the available knowledge, the system generates an answer 215. If the system determines that the available knowledge is not sufficient to answer the query 205, the system generates sub-queries that may help answer the query 205.

[0023] The system generates a plan for answering the question and executes the action phase 240. The system may execute code during the action phase 240. The code develops sub-queries that may provide an answer or help generate an answer. The system stores any additional information, such as the code used to answer the query 205. The stored information may then be used to answer additional queries. In this way, the system continues to build a domain-specific knowledge base that grows over time. The loop shown in Figures 1 and 2 is executed iteratively and may be executed multiple times.

[0024] The system may include a human in the loop for the orientation phase 220 and / or the decision phase 230 to approve the decision. However, in other embodiments, the system may perform the orientation phase 220 and / or the decision phase 230 automatically.

[0025] FIG. 3 illustrates an evaluation framework for determining whether a system can answer a query based on currently available information, according to an embodiment. According to an embodiment, the system includes an evaluation module for evaluating the system's capabilities. The system receives and stores a set of evaluation questions and answers for evaluating the system, which represent ground truth that can be used to determine whether the system can correctly answer these questions. These questions are domain-specific and test the knowledge or ability of the system (which represents an agent for answering domain-specific questions). The system answers the set of evaluation questions using a current language model and knowledge base. The system evaluates 320 the answers by comparing the generated answers with known answers in a ground truth data store. The system may use the LLM to compare the generated answers with ground truth answers stored in the system. For example, the system may generate a prompt including the generated answer and the ground truth answer and request the LLM to compare the two and provide a score indicating the accuracy of the generated answer. The system evaluates the system's knowledge based on the accuracy of the generated answers to the evaluation questions. If the system determines, based on the score, that the answer is satisfactory, the system determines that the system's available knowledge is sufficient to answer the query 205. If the generated answer is insufficient and does not approach the ground truth answer, the system determines that the system needs additional information to be able to answer the query 205. According to an embodiment, the system compares the score obtained by the system by answering the evaluation questions with a threshold. Based on the result of the comparison, the system determines whether the system's knowledge is sufficient to answer the query 205.

[0026] For example, in an industry-specific domain, an assessment question may ask the system to provide the types of materials that are suitable for a particular task. The ground truth answer lists materials known to be suitable for the task. The system generates the answer using information stored in the LLM and vector database. The system compares the materials identified by the system to the ground truth answer to determine whether the system correctly identified at least some materials, correctly identified all materials, or incorrectly identified no materials. The system scores the answer generated by the system for this question. The system scores all questions and generates an overall score that assesses the system's capabilities.

[0027] If the system determines that the answer is incorrect or insufficient, the system analyzes 330 the answer to determine why the answer was incorrect. For example, a question answer may include a series of steps or a list of items. If the generated answer includes only a subset of the steps in a subset of the items and skips some key steps / items, the system will score the result low.

[0028] The system may generate a solution by generating code, for example, Python code. The system may generate the Python code using LLM. If the system determines that the failure was due to a lack of information, the system may perform a search of various information stores for documents relevant to the query and store them in a vector database. The system repeats the complete OODA loop again based on the updated knowledge. Due to the system's increased knowledge, the system may improve its score. If the score is still not sufficient, the system may continue the procedure iteratively until the score improves to a satisfactory value.

[0029] Assessment questions may test knowledge of various processes. Assessment questions may test basic knowledge of domain-specific materials or equipment. Assessment questions may test knowledge of possible failures that can occur in a system. Assessment questions may check the system's ability to determine numerical errors, e.g., the existence of correct values ​​for parameters.

[0030] According to an embodiment, the system determines the following types of failures when evaluating the system: process failure; machine / maintenance failure; numerical error; lack of documented knowledge; lack of experiential learning / knowledge. The system may present the determined category or categories of causes of failure to the expert via a user interface. Alternatively, the system may automatically rank the available causes of failure and select the highest ranked cause.

[0031] The system can improve over days or even hours, updating itself to answer questions correctly. Additionally, the system can select a subset of knowledge or documents that are relevant to answering a particular domain-specific question. This prevents the system from unnecessarily storing large amounts of potentially unnecessary information. For example, an organization that prefers to keep proprietary information confidential may share minimal information with the system so that the system can answer questions with the minimum necessary information.

[0032] The system can be used to generate agents with domain-specific knowledge, where each agent has knowledge about a particular domain and no domain-specific knowledge about other domains. This allows the system to generate multiple domain-specific agents that compartmentalize knowledge rather than providing a single agent with all the available information for an organization. This allows the system to generate specialized agents with domain-specific knowledge.

[0033] (example) Below are the details of the OODA inference loop implemented by the system. The system is capable of answering complex domain-specific queries. For example, a user query could be, "Does XYZ telco have a reasonably healthy liquidity profile based on its quick ratio for FY2022, and if not, what other metrics are more relevant to measure its liquidity?"

[0034] The system implements the following phases: Tasks are systematically divided into subtasks across each phase.

[0035] (Observation phase) The main task in the observation phase is to make observations related to the user query, for example, the system evaluates the liquidity of XYZ Telecom Company using quick ratios for FY2022.

[0036] The system breaks this task down into subtasks that perform data extraction and preliminary calculations, for example, the system accesses and reviews XYZ's fiscal year 2022 financial statements and collects the required data (current assets, inventory, current liabilities). Quick Ratio = (Current Assets - Inventories) / Current Liabilities Using the first calculation approach, applying the formula: a Quick ratio of approximately 0.707 can be calculated.

[0037] The system may use a second calculation approach to calculate the quick ratio that focuses on the ratio of cash and cash equivalents plus accounts receivable and current liabilities, resulting in a ratio of 0.54, adjusting for the absence of marketable securities data.

[0038] (Inception Phase) In the Inception phase, the main task performed by the system is to analyze the calculated quick ratio to understand its impact on XYZ's liquidity. The system divides the main task into the following subtasks: (1) Compare Calculation Approaches: Review how each calculation method affects XYZ's perception of liquidity; and (2) Impact Assessment: Consider the significance of a quick ratio less than 1 and its impact on XYZ's ability to meet its short-term obligations.

[0039] (Decision phase) In the decision phase, the main task performed by the system is to make a decision about the health of XYZ's liquidity profile based on the quick ratio and its relevance as a metric. The system can divide the main task into the following subtasks: (1) Evaluating Liquidity Concerns: Evaluating whether the quick ratio suggests a healthy liquidity profile for XYZ; and (2) Considering Other Factors: Deciding whether the quick ratio alone can accurately reflect XYZ's financial health, or whether other indicators and considerations are necessary.

[0040] (Action Phase) In the action phase, the main task performed by the system is to draw conclusions about XYZ's liquidity profile and outline further considerations for a comprehensive analysis.

[0041] The system may divide the main task into the following subtasks: (1) Synthesis of findings: Combining observations and analysis into a final assessment of XYZ's liquidity; and (2) Identification of additional analysis needs: Highlighting the need for further analysis, including comparison with industry benchmarks, examining other liquidity metrics (e.g., current ratios, operating cash flow), and considering XYZ's long-term financial strategy.

[0042] This structured approach ensures a thorough and nuanced assessment of XYZ's liquidity profile, considering various aspects of its financial health and strategic positioning within its industry.

[0043] The implementation process for this advanced question-answering system begins with environment setup and data preparation, including detailed configuration and scripting to ensure seamless initial operation. The system design incorporates a sophisticated evaluation mechanism that leverages Python for data processing and analysis, providing a deep understanding of performance metrics and areas for improvement.

[0044] As development moves through phases of Observe, Inspire, Decide, and Act, each phase employs targeted Python scripts and classes designed to meticulously assess performance, identify failures, and devise viable solutions. This granular approach facilitates a nuanced understanding and addressing system inadequacies, ensuring each component functions optimally within the broader architecture.

[0045] The OODA methodology's incorporation of an iterative improvement loop fosters an environment of continuous evaluation and enhancement. This cycle of observe, orient, decision-making, and act forms the backbone of the system's adaptive capabilities, enabling iterative refinement that incrementally improves system performance to approach the level of accuracy of a human expert.

[0046] The system utilizes a structured approach from initial setup to final deployment, with an emphasis on rigorous testing, comprehensive documentation, and dedicated support, ensuring not only robust functionality of the system, but also adaptability and scalability to accurately and efficiently address the complex needs of industrial question-answering applications.

[0047] The system integrates domain-specific fine-tuning and iterative reasoning with search and expansion generation. It uses the OODA loop for continuous improvement and is highly adaptable, capable of generating answers at the level of a human expert. Benefits include improved accuracy, relevance, and adaptability in complex industrial environments, representing a significant leap over traditional question-answering systems. Its application across various industries has transformed information retrieval and decision-making processes, making it a versatile and valuable tool.

[0048] (Improved RAG-based system with improved accuracy) A system according to various embodiments uses an LLM to answer natural language-based questions from a user, e.g., natural language questions related to a specific domain, such as an industrial domain. The system is referred to as a retrieval-augmented generation (RAG)-based system. The system provides a set of documents, e.g., documents representing domain knowledge, for use in answering the questions. The documents may be stored in a document store, e.g., a vector database, and made available to the LLM. The document store may be referred to as a domain knowledge store. The system uses the LLM in combination with the knowledge stored in the document set to answer domain-specific questions. The accuracy of answers obtained by the system using the LLM depends on the domain knowledge stored in the document set available in the document store and on the prompts provided as input to the LLM. For example, the accuracy depends on whether the prompts provided by the system to the LLM contain all the information necessary to generate an answer to the natural language question received by the system. Conventional LLM-based systems are subject to hallucinations, as the LLM may produce answers that appear coherent and grammatically correct but are factually inaccurate or meaningless and may contain false or misleading information generated by the LLM. According to an embodiment, the system provides instructions to the LLM to not use any prior knowledge to answer the question and to rely only on documents available in a document store (domain knowledge store) to answer the question. The system provides additional knowledge (representing prior knowledge not available in the document store) to the LLM in prompts. For example, a natural language request may require knowledge of a formula or process for computing a result based on information stored in the document store. The system may obtain the relevant formula from an external source, for example, using a search engine, and provide the formula or computational process for computing the result along with instructions to the LLM to not use any prior knowledge the LLM has based on its training to compute the result.

[0049] 4 illustrates an exemplary RAG-based system based on LLM that provides improved accuracy of results, according to an embodiment. Other embodiments may include additional or fewer components than those shown in FIG.

[0050] The RAG-based system 410 may be an online system that receives natural language questions from users and answers the questions using a machine learning-based language model, e.g., a large-scale language model 430. The domain knowledge store 420 stores a set of documents that represent domain-specific knowledge, such as knowledge of a particular industry. According to an embodiment, the domain knowledge store 420 is a vector database that stores document embeddings and uses embedding-based search for documents that are within the vector distance of the natural language question. The RAG-based system 410 receives a natural language question 405 for answering. The RAG-based system 410 generates a vector representation of the natural language question 405. The RAG-based system 410 accesses the domain knowledge store 420 to identify a subset of documents relevant to the natural language question 405, e.g., a subset of documents having vector representations that are closest to the vector representation of the natural language question.

[0051] The RAG-based system 410 generates a prompt 415 for the large-scale language model 430 based on the natural language question 405. The RAG-based system 410 provides instructions to the prompt for the large-scale language model 430 that instruct the large-scale language model 430 not to use any prior knowledge that the large-scale language model 430 may have acquired during training to answer the natural language question 405. The RAG-based system 410 obtains any relevant prior knowledge that is not available in the domain knowledge store 420 from a prior knowledge source system 440. The RAG-based system 410 includes the prior knowledge obtained from the prior knowledge source system 440 in the prompt 415. The RAG-based system 410 provides the prompt 415 to the large-scale language model 430 for processing.

[0052] The RAG-based system 410 may further add prior knowledge obtained from the prior knowledge source system 440 to the domain knowledge store 420, so that the prior knowledge is immediately available from the domain knowledge store 420 for answering the subsequent natural language question 405 without having to access the prior knowledge source system 440. However, there may be other types of prior knowledge required for answering the subsequent natural language question 405 that may not yet be available in the domain knowledge store 420 and that require access from the prior knowledge source system 440 and be included in the corresponding prompt 415 generated for the large-scale language model 430 for answering the subsequent natural language question 405.

[0053] Conventional systems use domain-specific knowledge to retrain the large-scale language model 430 to improve the accuracy of answers generated by the large-scale language model 430. Accuracy may be measured using a series of questions that may test domain knowledge. Accuracy may represent the percentage of questions in the series that are correctly answered by the system. As shown in FIG. 4, the system disclosed herein provides a significantly higher level of accuracy compared to conventional systems based on retraining the large-scale language model 430 using domain knowledge. For example, experimentally, a system using the retrained large-scale language model 430 was observed to provide an approximately 10% improvement in accuracy on domain-specific questions, while the RAG-based system 410 shown in FIG. 4 was observed to improve accuracy by up to 30-40%. Therefore, instead of a system with high accuracy for a broad range of problems, the RAG-based system 410 shown in FIG. 4 provides much better accuracy when solving domain-specific problems. Such systems are more useful when dealing with domain-specific problems, such as industry-specific problems, which require high accuracy for domain-specific problems rather than high accuracy for broad problems requiring general knowledge. The system disclosed in FIG. 4 may be considered over-fitted for answering a general set of questions, but may be optimally adapted for a set of domain-specific questions.

[0054] According to an embodiment, the RAG-based system 410 identifies prior knowledge source systems for accessing prior knowledge information based on a machine learning-based language model.

[0055] According to an embodiment, the RAG-based system 410 generates another prompt requesting the machine learning-based language model to identify a particular prior knowledge source system that is expected to contain prior knowledge for resolving the natural language question. The RAG-based system provides another prompt to the machine learning-based language model and receives a second answer generated by running the machine learning-based language model based on the prompt. The RAG-based system 410 identifies a prior knowledge source system 440 from the second prompt.

[0056] According to an embodiment, the machine learning-based language model identifies one or more queries to access a prior knowledge source system to extract prior knowledge information necessary to answer the natural language question. For example, if the prior knowledge source system 440 is a database system, the RAG-based system 410 may request the large-scale language model 430 to generate database queries to access the prior knowledge information from the database system. If the prior knowledge source system 440 is identified as a search engine, the RAG-based system 410 may request the large-scale language model 430 to generate one or more search queries to access the prior knowledge information from the search engine.

[0057] 5 is a flowchart illustrating an overall process for answering domain-specific natural language questions using search expansion generation, according to an embodiment. The system stores a set of documents in a vector database, which may store domain-specific documents for a particular domain, for example, an industry domain from an industry such as the semiconductor industry, the oil and natural gas industry, or the manufacturing industry. The vector database stores a vector representation of each document.

[0058] The system receives a natural language question 510. The natural language question may be specific to a domain in which documents are stored in a vector database. The system generates a vector representation of the natural language question 520. The vector representation may be generated by providing the natural language question as input to a machine learning-based language model, representing an embedding of the natural language question.

[0059] The system retrieves a subset of documents relevant to the natural language question based on the vector representation of the natural language question 530. For example, the subset of documents may represent documents determined to be closest to the natural language question based on a distance metric representing a vector distance between the vector representation of the natural language question and each vector representation of the subset of documents.

[0060] The system determines prior knowledge information needed to add to the subset of documents retrieved from the vector database to answer the natural language question 540. The system identifies prior knowledge source systems for accessing the prior knowledge information 550. The system accesses the prior knowledge source systems to extract the prior knowledge information 560.

[0061] The system generates 570 a prompt for the machine learning-based language model that includes (1) a natural language question, (2) a subset of documents retrieved from the vector database, (3) prior knowledge information, and (4) instructions to the machine learning-based language model to avoid using the prior knowledge acquired by the machine learning-based language model during training of the machine learning-based language model. The system provides 580 the prompt to the machine learning-based language model and receives a response generated by running the machine learning-based language model based on the prompt.

[0062] The system performs an action based on the response. For example, the system may provide an answer to the user, for example, by sending the response to a client device for presentation to the user. Alternatively, the system may perform an automatic action based on the response. For example, in an industrial environment, the system may perform an action to control an industrial process, for example, by providing a particular signal to a control, by shutting down the system, or by throttling the system.

[0063] According to an embodiment, the system identifies a prior knowledge source system for accessing prior knowledge information based on the machine learning-based language model. Accordingly, the system generates a second prompt requesting the machine learning-based language model to identify a particular knowledge source system expected to contain prior knowledge for solving the natural language question, and provides the second prompt to the machine learning-based language model. The system receives a second response generated by running the machine learning-based language model based on the prompt, and identifies the prior knowledge source system from the second prompt.

[0064] According to an embodiment, the system adds one or more documents containing prior knowledge information to a vector database.

[0065] (APPLICATIONS) The system can be used in industries such as semiconductors, oil and gas, or manufacturing, where accuracy, efficiency, and decision-making speed are crucial. These sectors face complex challenges that require deep technical knowledge and rapid access to accurate information, making the system's domain-specific fine-tuning and iterative reasoning capabilities highly valuable. By improving the accuracy and adaptability of information retrieval and analysis, the system increases operational efficiency, reduces costs, and drives better decision-making.

[0066] (Computer Architecture) 6 is a high-level block diagram illustrating an exemplary system, according to an embodiment. Computer 600 includes at least one processor 602 coupled to a chipset 604. Chipset 604 includes a memory controller hub 620 and an input / output (I / O) controller hub 622. Memory 606 and a graphics adapter 612 are coupled to memory controller hub 620, and a display 618 is coupled to graphics adapter 612. Storage device 608, keyboard 610, pointing device 614, and network adapter 616 are coupled to I / O controller hub 622. Other embodiments of computer 600 have different architectures.

[0067] Storage device 608 is a non-transitory computer-readable storage medium such as a hard drive, compact disc read-only memory (CD-ROM), DVD, or solid-state memory device. Memory 606 holds instructions and data used by processor 602. Pointing device 614 is a mouse, trackball, or other type of pointing device and is used in combination with keyboard 610 to input data into computer system 600. Graphics adapter 612 displays images and other information on display 618. Network adapter 516 connects computer system 500 to one or more computer networks.

[0068] The computer 600 is adapted to execute computer program modules to provide the functionality described herein. As used herein, the term "module" refers to computer program logic utilized to provide a specified function. Thus, a module may be implemented in hardware, firmware, and / or software. In one embodiment, the program module is stored in the storage device 608, loaded into the memory 606, and executed by the processor 602. The type of computer 600 used may vary depending on the embodiment and requirements. For example, the computer may lack the display, keyboard, and / or other devices shown in FIG. 6.

[0069] (Other considerations) The disclosed embodiments improve the storage efficiency of time series data and the computational efficiency of time series data. Neural networks are useful for converting data sequences of any size into fixed-size feature vectors. In particular, input sequence data (or time series data) can be significantly larger than the feature vector representation generated by the hidden layer of a neural network. For example, the input time series may contain thousands of elements, while the feature vector representation of the sequence data may contain hundreds of elements. Thus, large sequences of data are converted into fixed-size and significantly smaller feature vectors. This provides an efficient storage representation of the sequence data. The storage representation can be for secondary storage, such as efficient storage on disk, or for in-memory processing or used for in-memory processing. For example, to process sequence data, a system with a given memory can handle a large number of feature vector representations of a sequence (compared to raw sequence data). Because many sequences can be loaded into memory simultaneously, data does not need to be written to secondary storage as frequently, making sequence processing more efficient.

[0070] Furthermore, the process of clustering sequences of data is significantly more efficient when performed based on the feature vector representation of the sequences compared to processing the sequence data itself. This is because the number of elements in the sequence data can be significantly larger than the number of elements in the feature vector representation of the sequences. Therefore, comparing the raw data of two sequences requires significantly more computations than comparing the two feature vector representations. Furthermore, because each sequence can be of a different size, comparing the data of two sequences will require additional processing to extract individual features.

[0071] Embodiments may perform neural network processing in parallel, for example, using a parallel / distributed architecture. For example, the operations of each node of the neural network may be performed in parallel, followed by a step of data communication between the nodes. Parallel processing of the neural network provides additional efficiency in the operation of the overall process described herein, for example, in FIG. 4.

[0072] It should be understood that the figures and descriptions of the present invention have been simplified to show elements relevant for a clear understanding of the present invention, but that for purposes of clarity, many other elements found in a typical distributed system have been omitted. Those skilled in the art may recognize that other elements and / or steps are desirable and / or necessary in implementing the embodiments. However, because such elements or steps are well known in the art and do not facilitate a better understanding of the embodiments, a discussion of such elements and steps is not provided herein. The disclosure herein covers all such variations and modifications to such elements and methods known to those skilled in the art.

[0073] Some of the above descriptions describe embodiments in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. While these operations are described functionally, operationally, or logically, they will be understood to be implemented by computer programs, or equivalent electrical circuits, microcode, or the like. Further, it has proven convenient at times to refer to arrangements of these operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0074] As used herein, a reference to "one embodiment" or "an embodiment" means that a particular element, feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

[0075] Some embodiments may be described using the terms "coupled" and "connected," along with their derivatives. It should be understood that these terms are not intended as synonyms for each other. For example, some embodiments may be described using the term "connected" to indicate that two or more elements are in direct physical or electrical contact with each other. In another example, some embodiments may be described using the term "coupled" to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" may also mean that two or more elements are not in direct contact with each other, but yet still cooperate or interact with each other. The embodiments are not limited in this context.

[0076] As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," or other variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus comprising a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent in such process, method, article, or apparatus. Furthermore, unless expressly stated otherwise, "or" refers to an inclusive "or," not an exclusive "or." For example, condition A or B can be satisfied by any one of the following: A is true (or present) and B is false (or absent), A is false (or absent) and B is true (or present), and both A and B are true (or present).

[0077] Furthermore, the use of "a" or "an" is employed herein to describe elements and components of embodiments. This is merely for convenience and to provide an overview of the invention. This description should be read to include one or at least one, and the singular also includes the plural unless it is clear that it is meant otherwise.

[0078] Upon reading this disclosure, those skilled in the art will recognize still additional alternative structural and functional designs for systems and processes for displaying charts using strain areas via the principles disclosed herein. Thus, while particular embodiments and applications have been illustrated and described, it should be understood that the disclosed embodiments are not limited to the precise structure and components disclosed herein. As will be apparent to those skilled in the art, various modifications, changes, and variations can be made in the arrangement, operation, and details of the methods and apparatus disclosed herein without departing from the spirit and scope, as defined in the appended claims.

Claims

1. 1. A computer-implemented method for search expansion generation of natural language questions and answers, comprising: storing a set of documents in a vector database, the vector database storing a vector representation of each of the documents in the set; receiving a natural language query; generating a vector representation of the natural language question; retrieving a subset of documents relevant to the natural language question based on the vector representation of the natural language question; determining prior knowledge information necessary to add to the subset of documents retrieved from the vector database to answer the natural language question; identifying a prior knowledge source system for accessing the prior knowledge information; accessing the prior knowledge source system to extract the prior knowledge information; generating a prompt for a machine learning based language model, comprising: the natural language query; a subset of the documents retrieved from the vector database; the prior knowledge information; instructing the machine learning based language model to avoid using prior knowledge acquired by the machine learning based language model during training of the machine learning based language model; providing the prompt to the machine learning based language model; receiving a response generated by executing the machine learning based language model based on the prompt; performing an action based on the response; and 11. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the vector database stores domain-specific documents for a particular domain.

3. The specific domain is semiconductor industry, the oil and natural gas industry, or manufacturing industry, The computer-implemented method of claim 2 , wherein the method represents one of the following industry domains:

4. 2. The computer-implemented method of claim 1, wherein the subset of documents represents documents from the set of documents that are determined to be closest to the natural language question based on a distance metric that represents a vector distance between the vector representation of the natural language question and the vector representation of each of the subset of documents.

5. The computer-implemented method of claim 1 , wherein identifying the prior knowledge source system for accessing the prior knowledge information is based on the machine learning based language model.

6. the prompt is a first prompt, the response is a first response, and identifying the prior knowledge source system for accessing the prior knowledge information includes: generating a second prompt requesting the machine learning based language model to identify a particular prior knowledge source system that is expected to contain prior knowledge for resolving the natural language question; providing the second prompt to the machine learning based language model; receiving a second response generated by executing the machine learning based language model based on the second prompt; identifying the prior knowledge source system from the second prompt; The computer-implemented method of claim 1 , comprising:

7. The computer-implemented method of claim 1 , further comprising: adding one or more documents containing the prior knowledge information to the vector database.

8. 1. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform steps for answer based search expansion generation of natural language questions, the steps comprising: storing a set of documents in a vector database, the vector database storing a vector representation of each of the documents in the set; receiving a natural language query; generating a vector representation of the natural language question; retrieving a subset of documents relevant to the natural language question based on the vector representation of the natural language question; determining prior knowledge information necessary to add to the subset of documents retrieved from the vector database to answer the natural language question; identifying a prior knowledge source system for accessing the prior knowledge information; accessing the prior knowledge source system to extract the prior knowledge information; generating a prompt for a machine learning based language model, comprising: the natural language query; a subset of the documents retrieved from the vector database; the prior knowledge information; instructing the machine learning based language model to avoid using prior knowledge acquired by the machine learning based language model during training of the machine learning based language model; providing the prompt to the machine learning based language model; receiving a response generated by executing the machine learning based language model based on the prompt; performing an action based on the response; and 1. A non-transitory computer-readable storage medium comprising:

9. The non-transitory computer-readable storage medium of claim 8 , wherein the vector database stores domain-specific documents for a particular domain.

10. The specific domain is Semiconductor industry, the oil and natural gas industry, or manufacturing industry, 10. The non-transitory computer-readable storage medium of claim 9, representing one of the industry domains:

11. 9. The non-transitory computer-readable storage medium of claim 8, wherein the subset of documents represents documents from the set of documents that are determined to be closest to the natural language question based on a distance metric that represents a vector distance between the vector representation of the natural language question and the vector representation of each of the subset of documents.

12. The non-transitory computer-readable storage medium of claim 8 , wherein identifying the prior knowledge source system for accessing the prior knowledge information is based on the machine learning based language model.

13. the prompt is a first prompt, the response is a first response, and identifying the prior knowledge source system for accessing the prior knowledge information includes: generating a second prompt requesting the machine learning based language model to identify a particular prior knowledge source system that is expected to contain prior knowledge for resolving the natural language question; providing the second prompt to the machine learning based language model; receiving a second response generated by executing the machine learning based language model based on the second prompt; identifying the prior knowledge source system from the second prompt; 9. The non-transitory computer-readable storage medium of claim 8, comprising:

14. adding one or more documents containing the prior knowledge information to the vector database; 10. The non-transitory computer-readable storage medium of claim 8, further comprising:

15. one or more processors; a non-transitory computer-readable storage medium storing instructions that, when executed by the one or more computer processors, cause the one or more computer processors to perform steps for answer based search expansion generation of natural language questions, the steps including: storing a set of documents in a vector database, the vector database storing a vector representation of each of the set of documents; receiving a natural language query; generating a vector representation of the natural language question; retrieving a subset of documents relevant to the natural language question based on the vector representation of the natural language question; determining prior knowledge information necessary to add to the subset of documents retrieved from the vector database to answer the natural language question; identifying a prior knowledge source system for accessing the prior knowledge information; accessing the prior knowledge source system to extract the prior knowledge information; generating a prompt for a machine learning based language model, comprising: the natural language query; a subset of the documents retrieved from the vector database; the prior knowledge information; instructing the machine learning based language model to avoid using prior knowledge acquired by the machine learning based language model during training of the machine learning based language model; providing the prompt to the machine learning based language model; receiving a response generated by executing the machine learning based language model based on the prompt; and performing an action based on the response. Computer system.

16. The computer system of claim 15 , wherein the vector database stores domain-specific documents for a particular domain.

17. The specific domain is semiconductor industry, the oil and natural gas industry, or manufacturing industry, 17. The computer system of claim 16, representing one of the industry domains:

18. 16. The computer system of claim 15, wherein the subset of documents represents documents from the set of documents that are determined to be closest to the natural language question based on a distance metric that represents a vector distance between the vector representation of the natural language question and the vector representation of each of the subset of documents.

19. The computer system of claim 15 , wherein identifying the prior knowledge source system for accessing the prior knowledge information is based on the machine learning based language model.

20. the prompt is a first prompt, the response is a first response, and identifying the prior knowledge source system for accessing the prior knowledge information includes: generating a second prompt requesting the machine learning based language model to identify a particular prior knowledge source system that is expected to contain prior knowledge for resolving the natural language question; providing the second prompt to the machine learning based language model; receiving a second response generated by executing the machine learning based language model based on the second prompt; identifying the prior knowledge source system from the second prompt; 16. The computer system of claim 15, comprising:

Images