Automated generation of a scoped and verified response from a large language model to a user query

Abstract

A method and system for application of a large language model to refine results from a query of a user in an organization is disclosed. A query related to an organization is received from the user. Context of the user is determined. A search of a database associated with the organization for factual information related to the query is conducted based on the determined context of the user. The relevance of the results of the search are determined. An input to a trained large language model is provided based on the determined relevant search results. A response to the query is generated via the trained large language model using the determined relevant search results.

Description

AUTOMATED GENERATION OF A SCOPED AND VERIFIED RESPONSE FROM A LARGE LANGUAGE MODEL TO A USER QUERYPRIORITY CLAIM

[0001] The present disclosure claims the benefit of and priority to U.S. Provisional Serial No. 63 / 580,873, filed on September 6, 2023. The contents of that application are hereby incorporated by reference in their entirety.TECHNICAL FIELD

[0002] The present disclosure generally relates to a system and method for providing a focused response to a user query by providing a user context based input to an LLM that allows more efficient, secure / private, and accurate responses from the LLM.BACKGROUND

[0003] Determining where there is an opportunity for productivity improvement in frontline workers or processes is an issue that conventional systems have attempted to solve unsuccessfully. For example, in conventional systems, procedures are simply available in paper or static electronic media without any ability to tailor the content of such procedures so workers may implement such procedures more efficiently. Further productivity improvements were performed through time and motion studies, i.e., watching a person do his or her job for a short period of time and then analyzing this data to identify those activities, steps, etc., that need to be changed to increase productivity. Such studies were often ineffective and time consuming. Newer systems enable more interactive guidance of frontline workers but require new procedures to be written in a different format.

[0004] Artificial intelligence such as large language models (LLMs) may be used to generate such guidance, but such systems require extensive computational resources to respond to different queries. In known LLMs, the user prompt that is a query from the user is passed unchanged into the LLM. The LLM uses the information in the user prompt to formulate the response to the query and thus searches an entire known knowledge base. This requires extensive computationalresources. Moreover, the responses from a general LLM may not be optimal for a particular worker procedure as the responses may incorporate irrelevant inputs,

[0005] Thus, there is a need for a method to ensure an automated and organization specific response to user inquiries based on using a LLM to generated a response to the inquiry. There is a need for providing an actionability score to evaluate the suitability of different possible responses to a query.SUMMARY

[0006] The term embodiment and like terms, e.g., implementation, configuration, aspect, example, and option, are intended to refer broadly to all of the subject matter of this disclosure and the claims below. Statements containing these terms should be understood not to limit the subject matter described herein or to limit the meaning or scope of the claims below. Embodiments of the present disclosure covered herein are defined by the claims below, not this summary. This summary is a high-level overview of various aspects of the disclosure and introduces some of the concepts that are further described in the Detailed Description section below. This summary is not intended to identify key or essential features of the claimed subject matter. This summary is also not intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this disclosure, any or all drawings, and each claim.

[0007] According to certain aspects of the present disclosure, a method for providing responses to queries is disclosed. A query related to an organization from a user is received. A context of the user is determined. A search of a database associated with the organization is conducted for factual information related to the query based on the determined context of the user. The relevance of the results of the search is determined. The query and an input based on the determined relevant search results are provided to a large language model. A response to the query is provided via the large language model using the determined relevant search results.

[0008] A further implementation of the example method is where the context of the user includes a subject of interest of the query. Another implementation is where the context of the user includes a level of access of the user in the organization. Another implementation is where the factual information includes at least one of instructions, procedures, manuals, lessons, training materials, peer generated comments, machine reading data, and user skill rating. Anotherimplementation is where the relevant search results are ranked, and the input of the relevant search results are limited by a maximum context input limit. Another implementation is where the example method includes displaying the output response. Another implementation is where the output response includes associated sources with the relevant results associated with the determined response. Another implementation is where the associated sources include a link allowing access to the database. Another implementation is where the associated sources include a link allowing communication with a person identified in the database. Another implementation is where the example method includes determining an actionability rating based on a skill of the user for each of the facts in the response.

[0009] According to certain other aspects of the present disclosure, a system for providing a response to a query is disclosed. The system includes a device generating a query related to an organization from a user. A database stores factual information associated with the organization. A processing module is coupled the device and the database. The processing module is operable to determine a context of the user and conduct a search of the database for factual information related to the query based on the determined context of the user. The processing module is operable determine the relevance of the results of the search. A large language model is coupled to the processing module. The large language model receives the query and an input based on the determined relevant search results and the large language model provides a response to the query.

[0010] A further implementation of the example system is where the context of the user includes a subject of interest of the query. Another implementation is where the context of the user includes a level of access of the user in the organization. Another implementation is where the factual information includes at least one of instructions, procedures, manuals, lessons, training materials, peer generated comments, machine reading data, and user skill rating. Another implementation is where the relevant search results are ranked, and the input of the relevant search results are limited by a maximum context input limit. Another implementation is where the example system includes a display displaying the output response. Another implementation is where the output response includes associated sources with the relevant results associated with the determined response. Another implementation is where the associated sources include a link allowing access to the database. Another implementation is where the associated sources include a link allowing communication with a person identified in the database. Another implementationis where the processing module is operable to determine an actionability rating based on a skill of the user for each of the facts in the response.

[0011] According to certain other aspects of the present disclosure, a non-transitory computer- readable medium having machine-readable instructions stored thereon is disclosed. The machine- readable instructions when executed by a processor, cause the processor to receive a query related to an organization from a user. The instructions cause the processor to determine a context of the user. The instructions cause the processor to conduct a search of a database associated with the organization for factual information related to the query based on the determined context of the user. The instructions cause the processor to determine the relevance of the results of the search. The instructions cause the processor to provide the query and an input based on the determined relevant search results to a large language model. The instructions cause the processor to generate a response to the query via the large language model using the determined relevant search results.DESCRIPTION OF DRAWINGS

[0012] So that the manner in which the above recited features of the present disclosure can be understood in detail, a more particular description of the disclosure, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrated only typical embodiments of this disclosure and are therefore not to be considered limiting of its scope, for the disclosure may admit to other equally effective embodiments.

[0013] FIG. l is a block diagram illustrating a computing environment, according to example embodiments.

[0014] FIG. 2 is a block diagram illustrating an analytics server, according to example embodiments.

[0015] FIG. 3 is a flow diagram of an example routine to respond to a user inquiry using a generative large language model.

[0016] FIG. 4 is a screen shot of a user entering a question into the example application user interface.

[0017] FIG. 5 shows a process of retrieving relevant facts the system performs to find the facts most relevant to the user’s question.

[0018] FIG. 6A shows an example of a question and an answer that are provided by the example application.

[0019] FIG. 6B shows another example of a question and an answer that are provided by the example application.

[0020] FIG. 6C shows an example instruction displayed from the answer provided in FIG. 6B.

[0021] FIG. 6D shows another example of a question and an answer that are provided by the example application.

[0022] FIG. 6E shows another example of a question and an answer that are provided by the example application.

[0023] FIG. 7A is a screenshot showing an answer to a user and additional user information available.

[0024] FIG. 7B is a screenshot showing a collaboration interface generated from a link in the answer provided in FIG. 7A.

[0025] FIG. 7C is a screenshot showing an answer to a user and identifying the supporting data source.

[0026] FIG. 8 is a process diagram of forming an input based on the example system for input to a LLM.DETAILED DESCRIPTION

[0027] Aspects of the invention will be apparent to those of ordinary skill in the art in view of the detailed description of various embodiments, which is made with reference to brief description provided herein.

[0028] FIG. 1 is a block diagram illustrating a computing environment 100, according to one embodiment. Computing environment 100 may include at least one or more worker devices 102, organization computing system 104, one or more author devices 108, one or more remote experts 110, a transactional database 124, and an insights database 126 communicating via network 105.

[0029] Network 105 may be of any suitable type, including individual connections via the Internet, such as cellular or Wi-Fi networks. In some embodiments, network 105 may connect terminals, services, and mobile devices using direct connections, such as radio frequency identification (RFID), near-field communication (NFC), Bluetooth™, low-energy Bluetooth™ (BLE), Wi-Fi™ ZigBee™, ambient backscatter communication (ABC) protocols, USB, WAN, orLAN. Because the information transmitted may be personal or confidential, security concerns may dictate one or more of these types of connection be encrypted or otherwise secured. In some embodiments, however, the information being transmitted may be less personal, and therefore, the network connections may be selected for convenience over security.

[0030] Network 105 may include any type of computer networking arrangement used to exchange data or information. For example, network 105 may be the Internet, a private data network, virtual private network using a public network and / or other suitable connection(s) that enables components in computing environment 100 to send and receive information between the components of system 100.

[0031] Author device 108 may be representative of computing devices, such as, but not limited to, a mobile device, a tablet, a desktop computer, or any computing system having the capabilities described herein. For example, author device 108 may be any device capable of executing software (e g., application 114) configured to author work procedures. Author device 108 may include application 114. In some embodiments, application 114 may be representative of a web browser that allows access to a web site. In some embodiments, application 114 may be representative of a stand-alone application. Author device 108 may access application 114 to access functionality of organization computing system 104. In some embodiments, author device 108 may communicate over network 104 to request a web page, for example, from web client application server 118 of organization computing system 104. The content that is displayed to author device 108 may be transmitted from web client application server 118 to author device 108, and subsequently processed by application 114 for display through a graphical user interface (GUI) of author device 108.

[0032] Author device 108 may be configured to execute application 118 to generate a new work procedure for assisting workers in performing a hands-on job, such as, but not limited to, equipment service, manufacturing assembly, or machine calibration. Exemplary work procedures may include any combination of text, pictures, movies, three-dimensional computer aided design (3D CAD), remote expert sessions, and mixed reality sessions to aid the worker in completing a task and tracking task completion.

[0033] Worker device 102 may be representative of a computer device, such as, but not limited to, a mobile device, a tablet, a desktop computer, a wearable device, smart glasses, or any computing system having the capabilities described herein. For example, worker device 102 maybe any device capable of executing software (e.g., application 112) configured to receive and display work procedures generated by an author device 108.

[0034] Worker device 102 may include application 112. In some embodiments, application 112 may be representative of a web browser that allows access to a website. In some embodiments, application 112 may be representative of a stand-alone application. Worker device 102 may execute application 112 to access functionality of organization computing system 104. In some embodiments, worker device 102 may communicate over network 105 to request a web page, for example, from web client application server 118 of organization computing system 104. For example, worker device 102 may request a work procedure from web client application server 118 for a particular hands-on job. The content that is displayed to worker device 102 may be transmitted from web client application server 118 to worker device 102, and subsequently processed by application 112 for display through a GUI of worker device 102. As will be explained below, worker device 102 may also be configured to send queries from a worker for assistance to the organization computing system 104. This query will be modified with worker context information and other data to receive a streamlined response from an LLM based assistance system.

[0035] In some embodiments, worker device 102 may be further configured to transmit activity data to organization computing system 104 for review and analysis. For example, via application 112, worker device 102 may transmit high granularity activity data to organization computing system 104, such that organization computing system 104 may analyze the activity data to improve (or optimize) the work procedure. Such high granularity activity data may include, but is not limited to, time it takes the worker to complete each step, whether the worker watched a video included in the work procedure, whether the worker stopped the video included in the work procedure early, whether the worker contacted a remote expert, what problems or questions were raised, whether the remote expert took any action, including making specific suggestions or recommendations, whether the user completed the work procedure, and the like.

[0036] Analyst device 106 may be representative of a computing device, such as, but not limited to, a mobile device, a tablet, a desktop computer, or any computing system having the capabilities described herein. For example, analyst device 106 may be any device capable of executing software (e.g., application 111) configured to access organization computing system 104.

[0037] Analyst device 106 may include application 1 11. In some embodiments, application 111 may be representative of a web browser that allows access to a website. In some embodiments, application 111 may be representative of a stand-alone application. Analyst device 106 may execute application 111 to access functionality of organization computing system 104. In some embodiments, analyst device 106 may communicate over network 105 to request a web page, for example, from web client application server 118 of organization computing system 104. The content that is displayed via analyst device 106 may be transmitted from web client application server 118 to analyst device 106, and subsequently processed by application 111 for display through GUI of analyst device 102. In some embodiments, analyst device 106 may act as the access point for data and visualization of that data along with generated insights and recommendations. The actions and responses of the analysts to these insights and recommendations, such as dismissing an insight and not taking any action on it. In some embodiments, actions performed by analyst devices 106 may be used as input to the machine learning model to better refine the model's generation of the insights.

[0038] Remote expert device 110 may be representative of a computing device, such as, but not limited to, a mobile device, a tablet, a desktop computer, or any computing system having the capabilities described herein. For example, remote expert device 110 may be any device capable of executing software (e.g., application 116) configured to access organization computing system 104.

[0039] Remote expert device 110 may include application 116. In some embodiments, application 116 may be representative of a web browser that allows access to a website. In some embodiments, application 116 may be representative of a stand-alone application. Remote expert device 110 may execute application 116 to access functionality or organization computing system. For example, remote expert device 110 may execute application 116 responsive to a request from a worker device 102 for guidance regarding an operation in a work procedure. Via application 116, through organization computing system 104, worker device 102 may be connected with a respective remote expert device 110. For example, in some embodiments, remote expert device 110 may communicate over network 105 to request a web page, for example, from web client application server 118 of organization computing system 104. The content that is displayed to remote expert device 110 may be transmitted from web client application server 118 to author remote expert device, 110 and subsequently processed by application 116 for display through aGUI of remote expert device 110. In other words, remote expert device 1 10 may enable a worker to stream live video and audio to remote experts. Both the expert and the worker may annotate either the live video or a freeze frame image. In some embodiments, remote expert device 110 may be presented with a complete history of the job that the worker is executing. The complete history may provide remote expert device 110 with the ability to “go back in time” to review the steps that were taken and the data that was input up to the time the remote expert session was established.

[0040] In some embodiments, remote expert device 110 may be further configured to transmit activity data to organization computing system 104 for review and analysis. For example, via application 116, remote expert device 110 may transmit high granularity activity data to organization computing system 104, such that organization computing system 104 may analyze the activity data to improve (or optimize) the work procedure, and to improve the selection of remote experts. Such high granularity activity data may include, but is not limited to, the assistance provided to worker device 102 during a respective work procedure, in the formats of audio, video, and text.

[0041] Organization computing system 104 may represent a computing platform configured to host a plurality of servers. In some embodiments, organization computing system 104 may be composed of several computing devices. For example, each computing device of organization computing system 104 may serve as a host for a cloud computing architecture, virtual machine, container, and the like.

[0042] Organization computing system 104 may include at least web client application server 118, analytics application programming interface (API) gateway 120, and analytics server 122. Web client application server 118 may be configured to host one or more webpages accessible to one or more of analyst device 106, author device 108, remote expert device 110, and / or worker device 102. In some embodiments, web client application server 118 may host one or more webpages that allow an author device 108 to generate a work procedure to be accessed by one or more worker device 102.

[0043] In another example, web client application server 118 may host one or more webpages that allow a worker device 102 to access one or more work procedures directed to the worker device 102.

[0044] Analytics server 122 may be configured to generate one or more actionable insights based on user activity data. For example, analytics server 122 may consume high resolution datagenerated by one or more of analyst device 106, author device 108, remote expert device 110, and worker device 102. From the high resolution data, analytics server 122 may be configured to generate one or more actionable insights related to a specific work procedure. Such insights may include, but are not limited to an author index (which may help users assess the needs of the authors and improve their skills, as well as better match their qualifications to the upcoming new tasks), dynamically improved (or optimized) and individualized work instructions for each worker device 102, a true opportunity score, a worker index, and the like. Analytics server 122 is described in more detail below in conjunction with FIG. 2.

[0045] Analytics API gateway 120 may be configured to act as an interface between web client application server 118 and analytics server 122. For example, analytics API gateway server 120 may be configured to transmit data collected by web client application server 118 to analytics server 122. Likewise, analytics API gateway server 120 may be configured to transmit insights generated by analytics server 122 to web client application server 118.

[0046] Analytics API gateway 120 may include API module 121. API module 121 may include one or more instructions to execute one or more APIs that provide various functionalities related to the operations of organization computing system 104. In some embodiments, API module 121 may include an API adapter that allows API module 121 to interface with and utilize enterprise APIs maintained by organization computing system 104 and / or an associated entity. In some embodiments, APIs may enable organization computing system 104 to communicate with one or more of worker device 102, analyst device 106, author device 108, remote expert device 110, or one or more third party devices.

[0047] In some embodiments, data collection and acquisition process provided by organization computing system 104 may include, but is not limited to: data from definition, modification and execution of new and existing procedures, data collected from external sources such as internet of things (loT) devices, customer relation management (CRM) devices, enterprise resource planning (ERP) devices by means of integrations provided to those systems, data collected from extended resources such as customers associated with organization computing system 104, data through human-in-the-loop feedback mechanism, which provides valuable domain expertise, and the like

[0048] Organization computing system 104 may communicate with transactional database 124 and insights database 126. Transactional database 124 may be configured to store raw data received (or retrieved) from one or more of worker devices 102, analyst devices 106, authordevices 108, remote expert devices 110, and the like. For example, web client application server 118 may be configured to receive data from one or more of worker devices 102, analyst devices 106, author devices 108, remote expert devices 110 and store that data in transactional database 124. Insights database 126 may be configured to store one or more actionable insights generated by analytics server 122. In operation, for example, analytics API gateway 120 may pull information from transactional database 124 and provide said information to analytics server 122 for further analysis. Upon generating one or more actionable insights via artificial intelligence and machine learning, analytics server 122 may be configured to store the generated insights in insights database 126. Analytics API gateway 120 may, in turn, pull the generated insights from insights database 126 and provide the insights to web client application server 118 for transmission to one or more of analyst device 106, author device 108, and / or worker device 110.

[0049] FIG. 2 is a block diagram illustrating analytics server 122 in greater detail, according to example embodiments. As illustrated, analytics server 122 may include pre-processing agent 202, authoring agent 204, job execution agent 206, job outcome agent 208, training agent 210, and operations management (OM) agent 212. Each of pre-processing agent 202, authoring agent 204, job execution agent 206, job outcome agent 208, training agent 210, and OM agent 212 may be comprised of one or more software modules. The one or more software modules may be collections of code or instructions stored on a media (e.g., memory of analytics server 118) that represent a series of machine instructions (e.g., program code) that implements one or more algorithmic steps. Such machine instructions may be the actual computer code the processor of analytics server 118 interprets to implement the instructions or, alternatively, may be a higher level of coding of the instructions that is interpreted to obtain the actual computer code. The one or more software modules may also include one or more hardware components. One or more aspects of an example algorithm may be performed by the hardware components (e.g., circuitry) itself, rather as a result of an instruction.

[0050] Pre-processing agent 202 may be configured to process data received from one or more of analyst device 106, author device 108, remote expert device 110, and / or client device 102. For example, pre-processing agent 202 may be configured to extract relevant information for the raw data and format it in a way that is compatible with an underlying algorithm, in which the data is used as input. In some embodiments, pre-processing agent 202 may identify up and down times of the work executions, of out of ordinary data points, remove irrelevant or noisy data points basedupon the feedback from analysts, and the like. In some embodiments, pre-processing agent 202 may further use data enhancement techniques such as, but not limited to, combining data points coming from different versions of work instructions based upon their similarity. Such prepared data may be stored in transactional database 124.

[0051] In some embodiments, pre-processing agent 202 may further be configured to extract one or more features from the prepared data. For example, pre-processing agent 202 may be configured to generate one or more data sets for each of authoring agent 204, job execution agent 206, job outcome agent 208, training agent 210, and OM agent 212 by extracting select features from the prepared data. Pre-processing agent 202 may include natural language processor (NLP) device 214. NLP device 214 may be configured to retrieve prepared data from transaction database 124 and scan the prepared data to learn and understand the content contained therein. NLP device 214 may then selectively identify portions of the prepared data that correspond to each respective agent. Based on this identification, pre-processing agent 202 may extract certain features for preparation of a data set for a particular agent.

[0052] Authoring agent 204 may be configured to improve the description of the set of activities that may lead to a desired outcome in a work procedure. As stated above, author devices 108 may be configured to define electronic work instructions (e.g., work procedures) in a highly structured, but flexible, way. Such functionality may allow authors to prescribe activities in self- contained chunks that correspond to major steps needed to accomplish an underlying end result. In some embodiments, the steps may be defined through a collection of different units of work (e g., “cards”). Each card may include, for example, a variety of different avenues for relaying instructions to a worker device 102. Such avenues may include, but are not limited to, text, table, checklist, images, videos, and the like.

[0053] One of the key problems in defining electronic work instructions is identifying which methodology provides the best (or optimal) content for the worker. Authoring agent 204 may be configured to aid in reducing the time it takes to reach an “optimal” procedure by providing insights to a work procedure through the use of the underlying procedure definition data, the experience representation (which may be built using historical data of successful work procedures) and correlations with execution patterns. For example, authoring agent 204 may be configured to generate actionable insights about an instruction's impact on the speed of execution, consistency of cycle time, and the quality of outcomes. Authoring agent 204 may generate a unique measurecalled “author index,” which may help organizations assess the needs of the authors and improve their skills, as well as better match their qualifications to upcoming tasks. Author index may represent a scoring mechanism to assess the process of authoring the electronic work instructions. In some embodiments, the score can be used to compare a given authoring process of a given work instructions for a specific product across other authoring processes of work instructions for similar products in order to identify the improvement opportunities for corresponding authors. This score may provide a benchmark as well as a measurement of goodness of such processes. For example, a given authoring process of a quality assurance (QA) procedure, which may include the number of versions it goes through, the count and types of different steps and cards, the time and quality indicators of the executed tasks can be used to create a score that would fairly compare that process against the authoring process of other similar procedures. This score value may provide insights into how the authoring process to proceed.

[0054] Authoring agent 204 may include machine learning module 216. Machine learning module 216 may include one or more instructions to train a prediction model to generate the author index described above. To train the prediction model, machine learning module 120 may receive, as input, one or more sets of training data generated by pre-processing agent 202. The training data utilized by authoring agent 204 may include but is not limited to: the versions of the instructions, the sequence of changes that are made to the instructions over time, number of steps, cards and their types, the way the instructions are expressed in terms of natural language and the style, the execution data in terms of cycle time, quality, and the sequence in which the work instructions are carried out as well as the patterns of up and down times, and the like. Machine learning module 216 may implement one or more machine learning algorithms to train the prediction model to generate the author index. Machine learning module 116 may use one or more of AB testing, a decision tree learning model, association rule learning model, artificial neural network model, deep learning model, inductive logic programming model, support vector machine model, clustering mode, Bayesian network model, reinforcement learning model, representational learning model, similarity and metric learning model, rule based machine learning model, and the like to train the prediction model.

[0055] Job execution agent 206 may be configured to generate one or more actionable insights for improving (or optimizing) work instructions for a given worker device 102. One of the main challenges in human-centric-processes is the inherent variability of the humans. Such variabilitymay place a huge strain on predictive systems that may be used for planning purposes. On the other hand, humans also provide flexibility to the system. For example, by employing workers rather than automated machines, a manufacturing organization may provide a huge variety of offerings. When author device 106, for example, generates a work procedure, the author may attempt to strike a balance between variability and flexibility. For example, such electronic work procedures may allow a user to standardize how the work should be accomplished, while also allowing for individualization as to how the instructions are delivered to respective worker devices 102.

[0056] Job execution agent 206 may be configured to continuously measure the “goodness” of the executions of underlying instructions. For example, job execution agent 206 may include machine learning module 218. Machine learning module 218 may include one or more instructions to train a prediction model to generate one or more actionable insights for improving work instructions for a given worker device 102. To train the prediction model, machine learning module 218 may receive, as input, one or more sets of training data generated by pre-processing agent 202. The training data utilized by authoring agent 204 may include up and down times, utilization of various media helpers, the order in which instructions are carried out, underlying experience and historical performance of the workers, worker's year of experience with the organization, the skill improvement training each worker has participated in, and the like. Machine learning module 218 may implement one or more machine learning algorithms to train the prediction model to generate a score associated with the quality of the instructions in a work procedure. For example, instead of machine learning module 218 may generate a quality of fit for a given worker is as a way to individualize the work instructions in such that they best serve the workers. Machine learning module 218 may use one or more, a decision tree learning model, association rule learning model, artificial neural network model, deep learning model, inductive logic programming model, support vector machine model, clustering mode, Bayesian network model, reinforcement learning model, representational learning model, similarity and metric learning model, rule based machine learning model, and the like to train the prediction model.

[0057] Based on the value generated by job execution agent 206, the system may be configured to provide dynamically optimized and individualized work instructions to enable each worker device to perform each time in the least amount of time possible, while achieving quality, safety, and productivity goals. In some embodiments, to achieve such dynamic optimization goal, thesystem may use techniques that involve, but are not limited to, exhibition or concealment of certain helper content, providing training refreshers, switching from verbose or succinct versions of the instructions, variations in the order in which steps are presented, and the like.

[0058] In some embodiments, job execution agent 206 may further provide unique insights into how the underlying worker execution fits with the set of identified goals, what kind of automatic interventions have been applied, and what kind of results have been attained from them. For example, consider a scenario in which five hundred units of the same product needs to be assembled for a given job order with certain time and quality targets. Job execution agent 206 may dynamically monitor the underlying executions and alter the details of the instructions provided to the user based upon that data. For example, given a worker establishing a good time average for the cycle time, he / she may no longer need the most detailed version of the instructions. Accordingly, job execution agent 206 may automatically make this switch. Similarly, a worker not establishing the required quality target can dynamically be provided with extended video instructions on the problematic step. In addition, the impact of these interventions may be provided to the stakeholders as insights for further training of workers or enhancement of work instructions.

[0059] Job outcome agent 208 may be configured to generate an opportunity value for a work procedure. One of the issues with human centric processes is the difficulty in collecting relevant data related to said human activities. Traditionally, conventional systems used time and motion studies, essentially watching a person do their job for a short period of time and then analyzing this data to identify what needed to be changed to increase productivity. Such techniques are typically expensive, intrusive to operations, and one-shot efforts. Job outcome agent 208 is configured to provide a unique solution that eliminates the downfalls of conventional systems. For example, through the use of granularity data transmitted by worker devices 102 to organization computing system 104, job outcome agent 208 may be configured to generate an opportunity value.

[0060] Job outcome agent 208 may include machine learning module 220. Machine learning module 220 may be configured to generate a raw opportunity score. For example, machine learning module 220 may include one or more instructions to train a prediction model to generate the raw opportunity score. To train the prediction model, machine learning module 220 may receive, as input, one or more sets of training data generated by pre-processing agent 202. The training data utilized by job outcome agent 208 may include, but is not limited to, time spent on productive andnon-productive sections of each card (or step), the information related to the identity of the worker device 102, status and quality of the underlying products that are being worked on, historical performance of the workers, the corresponding tools involved in the operation, and the like. Using this data, prediction model may be trained to identify a raw opportunity score.

[0061] From the raw opportunity score, job outcome agent 208 may generate a true opportunity score. For example, machine learning model 220 may further train the prediction model to generate the true opportunity using one or more machine learning algorithms. Machine learning module 220 may use one or more, a decision tree learning model, association rule learning model, artificial neural network model, deep learning model, inductive logic programming model, support vector machine model, clustering mode, Bayesian network model, reinforcement learning model, representational learning model, similarity and metric learning model, rule-based machine learning model, and the like to train the prediction model. The true opportunity score may be determined against a data driven benchmarks using techniques that involve, but are not limited to, identification and exclusion of noisy data points, adjustments taking into account the historical performance of the agent's quality, and resource state indicators.

[0062] In some embodiments, the true opportunity score may be continuously updated as more data is available to the system. Accordingly, job outcome agent 208 may provide a forward looking value that may be attained as a productivity improvement. Moreover, in some embodiments, job outcome agent 208 may qualify this score in terms of how much effort it would involve for the productivity improvement. This effort value may be attributed to various interventions that need to be done, such as, but not limited to, training required for the works or enhancements that should be made to the underlying procedure.

[0063] Training agent 210 may be configured to generate a worker index, which may quantify the learning needs and performance index of each worker (e.g., worker device 102). Generally, matching workers' skills to underlying tasks is a challenging process. Such process becomes convoluted due to the constant changes in the workforce and product requirements. For example, identifying, measuring, and meeting the learning needs of the workforce typically depends on a comprehensive evaluation of the individuals, the tasks, and the way in which the individuals are instructed to carry out those tasks.

[0064] Training agent 210 improves upon conventional systems by generating a multidimensional representation of the current state of the worker (e.g., associate / agent) in relationto the relative complexity of the activities involved in the current tasks that the given individual is assigned by making use of attributes of the worker and historical execution data collected by the platform. This representation of the worker may be used to quantify the learning needs and the performance index of the worker, i.e., the worker index. This score may constitute the basis for the continuous assessment of the training needs of each worker.

[0065] To generate the worker index, training agent 210 may include machine learning module 224. Machine learning module 224 may include one or more instructions to train a prediction model to generate the worker index described above. To train the prediction model, machine learning module 224 may receive, as input, one or more sets of training data generated by preprocessing agent 202. The training data utilized by machine learning module 224 may include, but is not limited to, attributes of each agent and historical execution data (e.g., including experience level for the underlying tools and the resources required). Machine learning module 224 may implement one or more machine learning algorithms to train the prediction model to generate the worker index for each worker device 102. Machine learning module 224 may use one or more, a decision tree learning model, association rule learning model, artificial neural network model, deep learning model, inductive logic programming model, support vector machine model, clustering mode, Bayesian network model, reinforcement learning model, representational learning model, similarity and metric learning model, rule based machine learning model, and the like to train the prediction model.

[0066] By continuous updating (or optimization) of the worker index, training agent 210 may provide guidance to the key stakeholders in terms of actionable insights with an intuitive interface, such that the workforce is supported by relevant training at the right time.

[0067] OM agent 212 may be configured to generate one or more decisions into prescriptive issues. For example, OM agent 212 may assign a set of works to an upcoming demand based on, for example, lead times, order quantities, resource utilization requirements, electronic work instructions, worker historical performances, and the like. Generally, organizations receive a greater benefit out of their improvement efforts when the entire system, as a whole, is considered. When goals are set at a high level, incorporating those constraints that connect multiple components increase the utility of the overall system. Through integrations with external data sources, such as enterprise resource planning, organization computing system 104 may be able to create a system level view. In some embodiments, the system level view takes into account theoperations as a whole in a given organization including various job orders, shared resources in terms of workers and equipment, as well as lead times of materials involved and the shipment dates. In some embodiments, the system level view may be created through connections to other systems the organization uses for those purposes such as ERP systems.

[0068] OM agent 212 may be configured to utilize this system level view to formulate key decisions into prescriptive problems. For example, the system level view may correspond to the data that enables the application to formulate such problems that improves (or optimizes) a goal function, subject to constraints that ties together the resources in general. An example of a key decision may be the assignment of workers to stations. The prescriptive problem that this decision corresponds may be to minimize the lead time or maximize the utilization of a certain equipment etc. For example, OM agent 212 may be configured to generate key decisions about workforce scheduling by taking into account demand and lead time constraints. Such key decisions may provide preventive maintenance decisions making use of the utilization and performance data through workers' cycle time. For example, data collected through IOT systems and ERP systems may be merged together with the frontline worker performance data, thereby providing input into determining when or if an equipment should undergo a preventive maintenance in order to avert a future outage.

[0069] OM agent 212 may include machine learning module 226. Machine learning module 222 may include one or more instructions to train a prediction model to generate the key decisions described above. To train the prediction model, machine learning module 226 may receive, as input, one or more sets of training data generated by pre-processing agent 202. The training data utilized by authoring agent 204 may include information collected from third party software that organizations may use to manage the operations that are not captured by organization computing system. Such data may be collected via TOT, ERP, or CRM systems and combined together with the proprietary data collected or generated by organization computing system 104, such as, but not limited to, cycle time, historical performance, resource utilization, etc. Machine learning module 226 may implement one or more machine learning algorithms to train the prediction model to generate the key decisions. Machine learning module 226 may use one or more, a large language model (LLM), a decision tree learning model, association rule learning model, artificial neural network model, deep learning model, inductive logic programming model, support vector machine model, clustering mode, Bayesian network model, reinforcement learning model, representationallearning model, similarity and metric learning model, rule-based machine learning model, and the like to train the prediction model.

[0070] In this example, the OM agent 212 also accepts queries from workers via computing devices such as the worker device 104 in FIG. 1. The OM agent 212 modifies the inputs to the LLM for efficient and relevant responses in response to the query from an LLM executed by the machine learning module 226.

[0071] The present disclosure relates to enhancing the application of a large language model (LLM) to provide results to a query of a user such as a worker operating the worker device 102. The response from the LLM is improved via providing a more reduced and limited input of data through an example LLM input modification system executed on the analytics server 122 in FIGs. 1-2. The input query is received and using the context of the user submitting the query, relevant facts are injected into the LLM. Thus, the example system limits the scope of information available to the LLM to fit the context of the user. This reduces the amount of data / tokens sent to the LLM speeding up the response (more efficient), and increases accuracy and security. The example system allows the response to the query to be reduced to more targeted responses by including the context of the user submitting the query before the query is submitted to the artificial intelligence module 226 executed by the OM agent 212 in FIG. 2. The refining of the input to the LLM allows incorporation of a more targeted set of facts derived from the context of the user in the LLM generated response.

[0072] The process for reducing the LLM input as executed by the example input modification system is as follows. A query related to an organization is received from the user via a device such as the worker device 102 and read by an example input modification system executed by the analytics server 122. The context of the user is determined based on a combination of stored data, such as personnel data, physical location, security permissions, etc., and dynamic and calculated dated such as machine status, Al-derived user proficiency, user actionability rating, etc., by the input modification system. The context of the user is defined as an intersection of their role (what they are allowed to do and see), their location in the organization (what equipment and processes are in scope / relevant), their actionability index (what actions are they authorized to take), and the process and equipment status associated with the user, at a particular moment in time. A search of a database associated with the organization for factual information related to the query is conducted based on the determined context of the user. The relevance of the results of the searchis determined. An input to a trained large language model is provided based on the determined relevant search results. A response to the query is generated via the trained large language model using the determined relevant search results. The refined input to the LLM results in faster processing and execution of the LLM as well as a more accurate and relevant answer to the user.

[0073] FIG. 3 is a flow diagram of an example routine to respond to a user inquiry using a generative large language model (LLM) based module. An input user inquiry is received (310). In this example, the input user inquiry may relate to work procedures, operating or trouble shooting a machine, or other queries specific to the organization. The user context is determined (312). The relevant facts are found based on the user context (314). An input is provided to the LLM relating to the user inquiry and the relevant facts (316). The LLM generates an answer to the received inquiry based on the input provided incorporating the relevant facts (318). The answer is filtered for an accountability match and provides a response to the user (320). The filtered response is then output to the user (322). In this example, the example answer summarizes the results and provides direct links to the sources in the internal database that the answer was derived from.

[0074] Once query from a user is received, the example input modification system determines the user context. In this example, the example input modification system determines the current context by collecting the probable subjects of interest from location and equipment data, the data access permissions of the user, and actionability rating. These determinations are made automatically without additional user input. The probable subjects of interest include specific training, relevant subject matter experts, work instructions, asset operating history, standard operating processes, operations and maintenance manuals, drawings, and the like. The probable subjects are determined based on the recent application navigation history and feature use by the user in relation to the analytics server 122. The user’s pre-defined data access policy covering which categories of subjects and specific instances of subjects the user is allowed to access. As an example a user in a junior operator role is allowed to view and execute startup equipment instructions but not perform troubleshooting procedures due to the safety risk related to their inexperience.

[0075] FIG. 4 is a screen shot of an example interface 400. The interface 400 has an inquiry window 410 that may be displayed to allow a user to enter a question into the example application user interface. The inquiry window 410 has a chat window 412. The system generates a prompt414 for the user. The user inquiry may be input through a user response window 416. The input may then be sent to the analytics system via a send button 418. As explained above, a modified input to an LLM is then generated based on the facts determined from user context.

[0076] Once the context is determined for the user, the example input modification system finds relevant facts relating to the user based on the context. The example input modification system searches a database such as the databases 124 or 126 for the factual information that the user is authorized to access. The example system determines the factual information that is the most relevant to the context of the user and the inquiry made by the user. The factual information is sourced from databases, APIs to cloud systems, and APIs to equipment control systems. An elastic search index is used to index all of the relevant data, which is then injected into the LLM.

[0077] Example factual information may include: a) work instructions or procedures; b) OEM maintenance manuals; c) One Point Lessons; d) Standard Operating Procedures; e) Training materials; f) Peer / user-generated content such as comments and conversations; g) Machine readings of machines operated by the user; h) Current job data values such as equipment status collected prior to the query; i) Past job history; j) Sy stem -generated scores and values such as True Proficiency or True Opportunity from the performance of the user; and k) Skill rating attainment based on organizational guidelines, and actionability index based on user role.

[0078] The facts are stored in an electronic database and are indexed and searchable by their values. For example, a user’s skill rating document may contain the following data fields: a) User ID - a system identifier for the user; b) User Name - user’s name; c) Skill ID - a system identifier for the skill; d) Skill Title - name of the skill; e) Skill Level Rating - a numerical rating on a predefined scale (e.g. between 0 and 4); and f) Subject - a label used to categorize the document by its subject matter

[0079] FIG. 5 shows a process of retrieving relevant facts the input modification system performs to find the facts most relevant to the user’s question before forwarding those facts to the LLM as an input. A database query is formulated (510). The query is formulated by combining the question text from the user and the information the system has retrieved about the user, including: the system identifier of the user; the organization identifier of the user; the most recently accessed subject of information (e.g. training, asset maintenance); and the most recently accessed asset information including its asset identifier. In the manufacturing field, assets are machines that make up production lines and the example user context provides the universe of assets that a userhas access to. The assets may be further refined to a subset of these assets based on where the physical location of the user, and where they are in the system when they make the query. This allows the query to automatically be enhanced by limitation to facts relevant to a single or subset of the assets.

[0080] The query is submitted to the electronic database (512). The processing of the database query is handled by the aforementioned electronic database (514). The results returned from the database query are ranked by their relevance to the submitted query (516). The top n number of results are selected as inputs for the LLM based on their degree of relevance and how likely they are to increase the quality of the answer generated by the LLM. The size of the selected facts as a whole is checked for length so that the facts plus the other parts of the LLM input do not exceed the maximum context input limit that can be accepted by the specific LLM.

[0081] FIGs. 6A-6D are example screen images showing example of a question and response provided by the example LLM module. Each of the example answers are provided based on a set of facts determined from the above described process of providing context based facts that are input into the LLM.

[0082] FIG. 6A shows an image 600 that includes an example question 602 and an example answer 604. The answer 604 includes a text answer 606 and various links 608. In this example, the question 602 that is “Who can perform centerline inspections?” The application provides the answer text 606 as: “Marcus Coulter [person], Supervisor - L5 [person], Level 2 Technician [person], and Level 1 Technician [person] can perform Centerline - Inspections [skill] at different levels.” The links 608 includes links to Marcus Coulter, a Supervisor-LS, a Level 2 Technician, a Level 1 Technician. In this example, clicking on the Marcus Coulter link would display a collaboration environment where the user can message Marcus Coulter, or start a live video call where they could collaborate on performing the Centerline inspection. Clicking on the other links, would display a collaboration environment or provide a call to the other individuals such as a Supervisor or Level 1 or Level 2 Technician in the organization.

[0083] FIG. 6B shows an image 620 that shows another example question 622 and an example response 624 that are provided by the example application. The answer 624 includes a text answer 626 and a link 628. In this example, the question 622 is “How to perform a 5s area audit (a procedure)?” The application provides the answer text 626 on a modified input to the LLM as: “Follow this procedure (on the screen) to conduct your 5s area audit. . .” The link 628 opens anelectronic Work Instruction that guides the user through an area audit and collects the required data.

[0084] FIG. 6C shows an example screen image 630 of the resulting work instruction displayed from clicking on the link 628 in FIG. 6B. As shown in FIG. 6B, the example work instruction returned by the LLM in the answer 624 is better targeted to the query. The work instruction in the screen image 630 includes a sort window 632 that includes a set of instructions and corresponding score fields that allows a user to enter a score for each task. A set in order window 634 includes another set of instructions and corresponding score fields that allows a user to enter a score for each task. In comparison, a normal answer from an LLM would return general information on 5S audits first then the user would have to refine the prompt “can you provide an answer for a packaging line” and it would return slightly more specific information, eventually the user would get something that might be of use, but not approved by the company and potentially not safe.

[0085] FIG. 6D shows an image 640 that shows another example question 642 and an example answer 644 that are provided by the example application. The answer 644 includes a text answer 646 and links 648. In this example, the question 642 entered by the user is “How to fix no data output (for an x-ray machine)?” The generated answer 644 is: “To solve this problem, go to the data output settings on your computer and set the number of repetitions to a relatively large number.” Additional links 648 includes a link for displaying an X-ray troubleshooting procedure and a link for an unverified post. The example application distinguishes between formally approved content and unverified content. Formally approved content is that content which has been verified to be truthful and can be used without safety or quality concerns. Unverified content may include be chat or messaging interactions between users that might solve the problem, but no one has reviewed and approved the content as be safe to use. An administrator has the option to not show unverified content in the answer.

[0086] FIG. 6D shows an image 640 that shows another example question 642 and an example response 644 that are provided by the example application. In this example, the question 642 is: “What are the open issues?” The answer 644 provided by the LLM is: “There are 7 open medium issues. There are 6 open critical issues. There are 1 open low issues.” In this example, there are no links because this is a report query from the user asking for information.

[0087] The generated and fdtered answer is displayed to the user in the system user interface as shown in FIGs. 6A-6E. As explained above, the answer identifies the supporting data source(s) (e.g. published procedures and unverified user posts) that are screened from the input data by the LLM. FIG. 7A is a screenshot showing the interface 400 where an answer to a user and additional user information available in the chat window 412. Thus, the answer 604 provided in FIG. 6A is shown displayed in the chat window 412.

[0088] FIG. 7B shows a video teleconference collaboration interface 710 that is generated by selecting a link to a person in the answer 604. In this example, the teleconference interface 710 may include a dialog window 712, a video window 714, and a sharing window 716. In this example, the sharing window 716 may be used to share input screens between the user and the person contacted via the link to display video or documents.

[0089] FIG. 7C is a screenshot showing a generated and filtered answer to a user in the system user interface 400. Thus, the answer 644 provided in FIG. 6D is shown displayed in the chat window 412.

[0090] In a typical LLM the question from the user prompt is passed into the LLM and the LLM uses the user prompt to formulate the response from all available data. In the example modified input system, the input to the LLM is enriched with a broad set of more relevant external data, including information about the user derived from the user context. The information about the user may include their physical location, security permissions, skill rating, certifications, etc. as well as sensor and machine data (e.g., is the machine running or not). The additional input information allows the LLM to efficiently return a more targeted and relevant response to the query.

[0091] Because user context is used in the example method to reduce the facts the LLM has to process the answer is returned in 10% of the time compared to an LLM with no reduction of facts. Also, the answer is more likely to be resolved in a single query. Traditional approaches return general information on the query and may require further queries.

[0092] In addition, an Actionability rating / scale is provided by the example modified input system to evaluate the results from the LLM. The example “Actionability Scale” provides a numeric value that determines how actionable each answer is. The Actionability Scale compares the skill / experience of the user to the skill / experience requirement to perform a task. For example, a junior operator can perform simple tasks such as “inspect the line” but not “clear the machinejam” since the machine jam requires experience in shutting down equipment and making sure there are no sources of stored energy. Thus, if a worker wants to clear a jammed piece of equipment and the LLM returns three different possible solutions with different actionability, and the actionability rating may guide which solution the worker can safely use. If a solution is a 10 rating, the user can employ the solution. If the actionability rating is a 3, the user probably cannot employe the solution. The customer / company also may determine what level of actionability that is passed to each worker based on their internal actionability scale and the actionability rating of the individual worker. This is based on a combination of role, skill level, and certification of the individual worker.

[0093] For example, in use cases that involve industrial equipment, users are trained and certified confirming they can safely and correctly perform specific processes for the equipment. The specific certifications for an industrial bottle filler could be: Start-up Filler, Shut-down Filler, Lock out / Tag out Filler, Changeover Filler to different bottle size, Clear Filler Jam. Certifications may range from:0 - cannot perform process1 - can perform process with Supervision2- Can perform independently3 - Can supervise others

[0094] Safe operation of equipment is a priority for industrial companies, so using the example LLM system to assist user to operate and troubleshoot equipment requires understanding of how “Actionable” the response is with respect to the capabilities of the user making the inquiry. Thus, the actionability determines what results may be given to a particular user to ensure they can use them in a safe manner.

[0095] As explained above, the example input modification system summarizes the results and provides direct links to the sources / facts / procedures / training, etc. that the answer was derived from. Each of the facts have an “actionability rating” that determines who is able to use this result to take action. Examples of different of “Actionability” ratings include:10 Anyone with a level 1 certification or above can use this result. For example, user who have attained Approved Work Instruction, reviewed Standard Operating Procedures, or reviewed troubleshooting guides7 Anyone with level 3 certification or a subject matter expert, or supervisor can use this result. Examples are users that have read the Machine Operations Manual or taken a Machine Training Course.5 Anyone with level 3 certification or a subject matter expert can use this result. Examples are a Machine Maintenance Manual, Collaboration Conversations, and the like.The example input modification system allows taking action during the prompt / response. For example, the system may notify a supervisor when a user wants to take action on a result that they need approval or supervision for.

[0096] FIG. 8 is an example flow diagram followed by the example input modification system to provide a reduced query to an LLM and return an answer. A prompt including inquiry is received (810). In this example, the worker or user enters the user prompt through the worker device 102 on an interface such as that shown in FIG. 4. In this example, the query is “how can I clear a jam on the filler machine.” The example input modification system enriches the prompt with a wide range of common and specialized inputs collected based on the context of the user to then select relevant “facts.” In this example, the common and specialized inputs (820) include Security / Permission level; user Role; Organizational location; Equipment ID(s); Skill Level; Compliance Status; “true proficiency” level; User Actionability scale rating; Machine status; IOT data; Operational Recency; and Time in Position

[0097] The example system selects certain facts from the fact universe based on a query to the database from the context of the user (830). Work Instruction 1 (e.g., Centerlining the 6 position Filler); Work instruction 2 (Operations of the 6 position Filler); standard operating procedures (Filler Startup, Filler Shutdown N); One Point Less (a one page sheet to educate users on a specific task e.g., “disposing of sharp objects” (OPL1 - OPL N); a Filler Manual; a Filler Troubleshooting Instruction a Filler training video; Collaboration Threads (e.g., Collaboration Thread 1 to Collaboration Thread N); Filler CAD Drawings; Filler Operations Skill (the training to perform a Filler procedure - e.g., Filler Start-up training), and the like.

[0098] In this example, the relevant facts are work instruction 1, SOP N, OPL N, Filler troubleshooting guide, the Filler training video, and the Filler operations skill. The relevant facts are selected based on the context of the user from the database.

[0099] The resultant “facts” are injected as an input into the LLM 840 with the query. Based on the query and the injected facts, the LLM 840 returns a response to the query with asummarization of the content and the facts that are the most applicable to the prompt (850). In this example, the facts returned are Work Instruction 1, OPL N, filler trouble shooting and Filler training video. These are ordered based on LLM ranking and fed into the example input modification system.

[0100] The example input modification system orders the responses by the actionability rating. In this example, the responses are ordered first by those that the user can execute independently, second by the most relevant, and third by those that require a prior approval or direct supervision, input priorities may be provided, and the order may be changed by an administrator. The example input modification system can take appropriate action to enable these responses via notification or calling a custom function.

[0101] A response to the user from the prompt is then formulated and rendered on a display for the user (870). In this example, the responses from the data are ordered and helpful language is inserted in the response. Thus, the example response includes the two responses that don’t require a supervisor and links 872 and 874 that allow the user to access the responses from the database. The answer also includes the response that had the highest accountability. In this example, the answer includes a link 876 that allows the user to contact a supervisor to use Filler troubleshooting.

[0102] The LLM 840 may be leveraged multiple times in the process as well, in order to improve the response.

[0103] Additional details for the above-mentioned principles may be disclosed in U.S. Patent Application Serial Nos. 16 / 849,744 and 17 / 251,723, the contents of which are hereby incorporated by reference.

[0104] Although the disclosed embodiments have been illustrated and described with respect to one or more implementations, equivalent alterations and modifications will occur or be known to others skilled in the art upon the reading and understanding of this specification and the annexed drawings. In addition, while a particular feature of the invention may have been disclosed with respect to only one of several implementations, such feature may be combined with one or more other features of the other implementations as may be desired and advantageous for any given or particular application.

[0105] While various embodiments of the present disclosure have been described above, it should be understood that they have been presented by way of example only, and not limitation.Numerous changes to the disclosed embodiments can be made in accordance with the disclosure herein, without departing from the spirit or scope of the disclosure. Thus, the breadth and scope of the present disclosure should not be limited by any of the above described embodiments. Rather, the scope of the disclosure should be defined in accordance with the following claims and their equivalents.

Claims

WHAT IS CLAIMED IS:

1. A method for providing responses to queries, comprising: receiving a query related to an organization from a user; determining a context of the user; conducting a search of a database associated with the organization for factual information related to the query based on the determined context of the user; determining the relevance of the results of the search; providing the query and an input based on the determined relevant search results to a large language model; and generating a response to the query via the large language model using the determined relevant search results.

2. The method of claim 1, wherein the context of the user includes a subject of interest of the query.

3. The method of claim 1, wherein the context of the user includes a level of access of the user in the organization.

4. The method of claim 1, wherein the factual information includes at least one of instructions, procedures, manuals, lessons, training materials, peer generated comments, machine reading data, and user skill rating.

5. The method of claim 1, wherein the relevant search results are ranked, and wherein the input of the relevant search results are limited by a maximum context input limit.

6. The method of claim 1, further comprising displaying the output response.

7. The method of claim 5, wherein the output response includes associated sources with the relevant results associated with the determined response.

8. The method of claim 7 , wherein the associated sources include a link allowing access to the database.

9. The method of claim 7, wherein the associated sources include a link allowing communication with a person identified in the database.

10. The method of claim 1, further comprising determining an actionability rating based on a skill of the user for each of the facts in the response.

11. A system for providing a response to a query comprising: a device generating a query related to an organization from a user; a database storing factual information associated with the organization; a processing module coupled the device and the database, operable to: determine a context of the user; conduct a search of the database for factual information related to the query based on the determined context of the user; and determine the relevance of the results of the search; and a large language model coupled to the processing module, the large language model receiving the query and an input based on the determined relevant search results and the large language model providing a response to the query.

12. The system of claim 11, wherein the context of the user includes a subject of interest of the query.

13. The system of claim 11, wherein the context of the user includes a level of access of the user in the organization.

14. The system of claim 11, wherein the factual information includes at least one of instructions, procedures, manuals, lessons, training materials, peer generated comments, machine reading data, and user skill rating.

15. The system of claim 1 1 , wherein the relevant search results are ranked, and wherein the input of the relevant search results are limited by a maximum context input limit.

16. The system of claim 11, further comprising a display that displays the output response.

17. The system of claim 15, wherein the output resource includes associated sources with the relevant results associated with the determined response.

18. The system of claim 11, wherein the associated sources include a link allowing access to the database.

19. The system of claim 17, wherein the associated sources include a link allowing communication with a person identified in the database.

20. The system of claim 11, wherein the processing module is further operable to determine an actionability rating based on a skill of the user for each of the facts in the response.

21. A non-transitory computer-readable medium having machine-readable instructions stored thereon, which when executed by a processor, cause the processor to: receive a query related to an organization from a user; determine a context of the user; conduct a search of a database associated with the organization for factual information related to the query based on the determined context of the user; determine the relevance of the results of the search; provide the query and an input based on the determined relevant search results to a large language model; and generate a response to the query via the large language model using the determined relevant search results.

Images