Generative ai appliance

Abstract

In example implementations described herein, there are systems and methods for generating output associated with an industrial application including extracting content and context information associated with the industrial application, producing one or more context- specific large language models (LLMs) by adjusting one or more LLMs based on the content and context information, identifying a triggering event associated with a particular context; and generating a response to the triggering event using a context-specific LLM corresponding to the particular context.

Description

GENERATIVE Al APPLIANCEBACKGROUNDField

[0001] The present disclosure is generally directed to a digital advisor associated with an industrial application.Related Art

[0002] A generative pre-trained transformer (GPT) may be used to provide natural language answers or recommendations based on user input (e.g., a natural language query). However, training a GPT (or similar generative artificial intelligence (Al) systems or appliances), in some cases, requires large amounts of resources. Using a generic GPT (e.g., a GPT trained based on publicly available resources and / or documents) may be sufficient for answering general knowledge questions, but may not be useful in addressing context-specific queries. Accordingly, methods for training and deploying context-aware (or context-specific) Generative Al systems and / or appliances may be needed to address context-specific queries (e.g. to provide answers or recommendations that are usefill in a specific context).SUMMARY

[0003] Example implementations described herein involve an innovative method to address diverse challenges within industrial sectors using a Generative Al appliance having natural language processing (NLP) abilities. The Generative Al appliance (e.g., a Generative Al companion or Generative Al companion system) may offer recommendations for actions, configurations, services, or content based on user preferences, and / or prior system settings. This time-saving approach not only streamlines user actions but also aids in the discovery of new impactful events.

[0004] For example, in industrial applications, Generative Al companions may automate tasks like proposing system variable settings, aligning software parameters with hardware parameters, or executing office automation tasks such as sending alerts. This enhances efficiency, minimizing user effort while boosting productivity. Generative Al companionsystems, in some aspects, may engage in meaningful conversations with users. This aids in translating intricate configurations into user-friendly contexts and extends to providing monitoring and troubleshooting support. In some aspects, the Generative Al companions may facilitate user adoption of new industrial application settings and novel applications. For example, the Generative Al companions may offer tailored insights through user-friendly conversations, guiding users on ’’What" and "How” related to their backgrounds.

[0005] An element of Generative Al appliances is in-context learning. In-context learning, in some aspects, may involve the development of customized Generative Al appliances that seamlessly integrate with Business Applications through their user interface (UI) and / or user experience (UX). One challenge of in-context learning involves the collection of contextspecific data for businesses. The training of Generative Al using this (collected) data to generate precise, valuable, and actionable outcomes can also present certain complexities. If these challenges are overcome, the result, in some aspects, may be the creation of more streamlined and user-friendly appliances tailored to distinct business and operational scenarios. This disclosure relates to a unified appliances platform with adaptable contexts achieved through a Generative Al verticalization fine-tuning approach, ultimately enhancing Generative Al system and its management to cater to a range of diverse requirements.

[0006] Aspects of the present disclosure include a method for generating output associated with an industrial application. The method may include extracting content and context information associated with the industrial application; producing one or more context-specific large language models (LLMs) by adjusting one or more LLMs based on the content and context information; identifying a triggering event associated with a particular context; and generating a response to the triggering event using a context-specific LLM corresponding to the particular context.

[0007] Aspects of the present disclosure include a non-transitory computer readable medium, storing instructions for execution by a processor, which can involve instructions for extracting content and context information associated with the industrial application; producing one or more context-specific LLMs by adjusting one or more LLMs based on the content and context information; identifying a triggering event associated with a particular context; and generating a response to the triggering event using a context-specific LLM corresponding to the particular context.

[0008] Aspects of the present disclosure include a system, which can involve means for extracting content and context information associated with the industrial application; producing one or more context-specific LLMs by adjusting one or more LLMs based on the content and context information; identifying a triggering event associated with a particular context; and generating a response to the triggering event using a context-specific LLM corresponding to the particular context.

[0009] Aspects of the present disclosure include an apparatus, which can involve a processor, configured to extract content and context information associated with the industrial application; produce one or more context-specific LLMs by adjusting one or more LLMs based on the content and context information; identify a triggering event associated with a particular context; and generate a response to the triggering event using a context-specific LLM corresponding to the particular context.BRIEF DESCRIPTION OF DRAWINGS

[0010] FIG. 1 is a diagram illustrating components of a Generative Al system in accordance with some aspects of the disclosure.

[0011] FIG. 2 is a flow diagram illustrating a method associated with the interactive application analysis and knowledge extraction performed by a Generative Al system in accordance with some aspects.

[0012] FIG. 3 is a diagram illustrating operations associated with generating the (dynamic) knowledge graph and / or in-context tree associated with an application in accordance with some aspects of the disclosure.

[0013] FIG. 4 is a diagram illustrating the extraction of content from a UI / UX associated with an application in accordance with some aspects of the disclosure.

[0014] FIG. 5 is a diagram illustrating the elements of a knowledge graph based on the incontext tree of FIG. 4 in accordance with some aspects of the disclosure.

[0015] FIG. 6 is a diagram illustrating an additional example of content extraction from a screenshot in accordance with some aspects of the disclosure.

[0016] FIG. 7 is a diagram illustrating aspects of content and / or context extraction and analysis in accordance with some aspects of the disclosure.

[0017] FIG. 8 illustrates a set of components of the system associated with training the system in accordance with some aspects of the disclosure.

[0018] FIG. 9 is a diagram illustrating components of the system associated with an inferencing operation in accordance with some aspects of the disclosure.

[0019] FIG. 10 is a diagram illustrating a first embedding technique in accordance with some aspects of the disclosure.

[0020] FIG. 11 is a diagram illustrating a second embedding technique in accordance with some aspects of the disclosure.

[0021] FIG. 12 is a diagram illustrating a third embedding technique in accordance with some aspects of the disclosure.

[0022] FIG. 13 is a diagram illustrating a high-level view of the operations discussed above in relations to FIGs. 1-12 in accordance with some aspects of the disclosure.

[0023] FIG. 14 is a diagram illustrating components of the system associated with training a question and answer generator in accordance with some aspects of the disclosure.

[0024] FIG. 15 is a diagram illustrating a set of elements relating to the training of a question generator (QG) in accordance with some aspects of the disclosure.

[0025] FIG. 16 is a diagram illustrating a set of elements relating to the training of am answer generator (AG) in accordance with some aspects of the disclosure.

[0026] FIG. 17 is a diagram illustrating a method associated with the use of a framework associated with the system in accordance with some aspects of the disclosure.

[0027] FIG. 18 is an illustration of a set of operations associated with a conversational UI to generate an answer for a question and / or prompt in accordance with some aspects of the disclosure.

[0028] FIG. 19 is a diagram illustrating a set of elements of a reinforcement learning associated with the system in accordance with some aspects of the disclosure.

[0029] FIG. 20 is a flow diagram illustrating a method of generating output associated with an industrial application in accordance with some aspects of the disclosure.

[0030] FIG. 21 is a flow diagram illustrating a method of generating output associated with an industrial application in accordance with some aspects of the disclosure.

[0031] FIG. 22 illustrates an example computing environment with an example computer device suitable for use in some example implementations.DETAILED DESCRIPTION

[0032] The following detailed description provides details of the figures and example implementations of the present application. Reference numerals and descriptions of redundant elements between figures are omitted for clarity. Terms used throughout the description are provided as examples and are not intended to be limiting. For example, the use of the term “automatic" may involve fully automatic or semi-automatic implementations involving user or administrator control over certain aspects of the implementation, depending on the desired implementation of one of the ordinary skills in the art practicing implementations of the present application. Selection can be conducted by a user through a user interface or other input means, or can be implemented through a desired algorithm. Example implementations as described herein can be utilized either singularly or in combination and the functionality of the example implementations can be implemented through any means according to the desired implementations.

[0033] Some aspects of the disclosure relate to a Digital Advisor, a Content Al, and / or a Common Software Foundation. A Generative Al Companion Appliance, in some aspects, may be an appliance that uses Generative Al to interact with users and provide them with a more personalized and engaging experience. These appliances can learn the user’s preferences and habits, and then use that information to provide personalized recommendations, create custom content, or simply have more natural and engaging conversations

[0034] The disclosure related to developing and standardizing the Al Companion software components for Generative Al Appliance implementation, the primary design for on-premise servers, relevant Industrial applications, Application UI / UX topics extraction and knowledge graph formulation, and optimizing usability of user journey and system performance with userover time. Components and / or functions discussed herein include LLMs, e.g., GPT, a knowledge management system and blueprint, knowledge topics and properties creation, fine tuning question and answer generators, a knowledge content management system, a data catalog, model management, and workflow orchestration.

[0035] Al -powered companion appliance interacting with industrial software application UI / UX, in some aspects, may offer benefits for businesses. An Al-powered appliance as a companion system, has become increasingly useful in digital space with human and Al integration. Some of the reasons relate to traditional areas of concerns, e.g., optimizing one or more of human efficiency, task scalability, and / or task personalization. For example, an Al companion, in some aspects, may be used to automate tasks and improve the efficiency of industrial software applications when interacting with a user (e.g., a human user of software agent) . For example the (human or software agent) user may be collaborating with a Generative Al assistant such that the user is actively engaged in the task at hand, and the Al assistant is providing support and assistance. In some aspects, a (human or software agent) user may leverage the Generative Al to be more productive and efficient such that the Al assistant may be automating tasks, providing insights, and helping the user to make better decisions. A (human or software agent) user, in some aspects, may be in a symbiotic relationship with Al such that the user and the Al assistant are working together to achieve a common goal, e.g. a navigation and mobility Gen Al assistant for a blind person. Al companion can be used to automatically generate reports, diagnose problems, and optimize processes. This can free up human workers to focus on more strategic tasks. In some aspects, an Al companion may be used to help industrial workers be more productive. For example, an Al companion can be used to provide real-time feedback, suggest solutions, and automate tasks in the application UI / UX. This can help workers complete their tasks faster and more accurately.

[0036] In some aspects, an Al companion can be used to help industrial workers make better decisions. For example, Al can be used to analyze data, identify trends, and provide recommendations from UI / UX. This can help workers make better decisions about production, scheduling, and resource allocation. The Al companion, in some aspects, may be used to reduce the costs of industrial software applications to address issue identified. In some aspects, the Al companion can be used to automate tasks, which can reduce the need for human workers. The Al companion, in some aspects, may also be used to optimize processes, which can reduce the amount of energy and materials used.

[0037] The Al companion, in some aspects, may be used to improve safety in industrial settings. With information regarding a context relating to an application, an Al companion may be used to monitor equipment status, detect hazards, and issue warnings (contextual information). This may help to prevent accidents and injuries. Additionally, Generative Al companion systems, in some aspects, may be more engaging than traditional appliances, as they can have more natural and interactive conversations with the user. This may make the appliance more enjoyable to use.

[0038] Furthermore, Generative Al companion systems, in some aspects, may learn from the user’s interactions with them. This can help the appliance improve its performance over time and become more personalized to the user. The process of in-context learning, in some aspects, entails the development of customized Generative Al appliances that seamlessly integrate with Business Applications through their UI / UX. As described above, a challenge associated with in- context learning involves the collection of context-specific data for businesses. The training of Generative Al using this data to generate precise, valuable, and actionable outcomes, in some aspects, may also present certain complexities. However, if these challenges are overcome as described below, the outcome may be the creation of more streamlined and user-friendly appliances tailored to distinct business and operational scenarios. Accordingly, this disclosure relates to a unified appliances platform with adaptable contexts achieved through a Generative Al verticalization fine-tuning approach, ultimately enhancing a Generative Al system and its management to cater to a range of diverse requirements.

[0039] FIG. 1 is a diagram 100 illustrating components of a Generative Al system in accordance with some aspects of the disclosure. The components illustrated in FIG. 1, in some aspects, may be conceptually grouped into a set of functions and / or subsystems (where some components may be associated with multiple functions and / or subsystems). A first application and application context extraction subsystem may be related to providing an application 110 via UI / UX 111 and extracting context related to the application via Generative Al UI / UX (Gen Al UI / UX 112). The first application and application context extraction subsystem, in some aspects, may involve identifying the characteristics of an application, as well as the specific business and operational contexts) in which it operates.[0040 J A second subsystem may relate to Generative Al based user interactions and may, in some aspects, involve leveraging the capabilities of Generative Al models to enhance and facilitate interactions between users within the application’s UI / UX. The second subsystem, insome aspects, may utilize Al-generated content, responses, and / or suggestions to augment a user’s experiences. In some aspects, Generative Al based user interactions may be used to generate new content, such as text and / or instruction sequences responding to in-context questions and to produce contextually relevant outputs. In some aspects, the user interactions may include providing one or more user prompts 122.

[0041] A third subsystem may relate to context extraction. The context extraction may involve the process of identifying the context of a word or phrase in a document. In some aspects, the context extraction may use an in-context tree and knowledge graph (e.g., application operational knowledge graph 151, application business knowledge graph 161, application knowledge subgraph 131). The in-context tree, in some aspects, may indicate and / or illustrate the relationships between words in a document and can be used to identify the context of a word or phrase by looking at the words that are connected to it. The knowledge graph, in some aspects, may be used to provide additional context for the words in a document. This is done by connecting entities (UIs or pages associated with the application 110) and their relationships (e.g., connections via navigation through the apphcation UI / UX 111 associated with the application 110). By using the in-context tree and knowledge graph, extracting the context of a word or phrase in a document may be more accurate.

[0042] A fourth system may relate to operational context and business context verticalization (or fine tuning) of LLMs to process (adjust) and / or adapt the LLM to a specific domain, such as finance, healthcare, or customer service. This can be done by providing the LLM with data that is specific to the domain, and the operational appliance that support this business. The LLM can then be fine-tuned to learn the patterns and relationships in this data allowing it to perform better on tasks that are relevant to the domain and surrounding infrastructure construct.

[0043] In order to fine-tune the LLM to meet the specific needs of businesses in a Generative Al appliance settings, in some aspects, the Generative Al system may use one of three embedding variants (e.g., common embeddings 132 or 143) for LLMs in accordance with the disclosure. These embeddings combine three types of embeddings: a word embedding, a position embedding, and a node embedding. Word embeddings, in some aspects, may represent the meaning of individual words. In some aspects, position embeddings may represent the position of a word in a sentence. Node embeddings, in some aspects, may represent the relationships between words in a sentence. By combining these three types of embeddings (inone of three ways discussed below), a more comprehensive and accurate representation of words and phrases may be created. This improved representation may be used to improve the performance of (fine-tuned and / or context-specific) LLMs on a variety of tasks. This is beneficial for both the operation of the appliance and the business processes that it supports.

[0044] A fifth system may relate to an LLM Question and Answer Generator (LLM-QA) (e.g., one or more of question and answer generator 155 or question and answer generator 165). The LLM-QA, in some aspects, may be an automated approach that can be used to generate answers to questions. It can be further improved by fine-tuning an initial LLM model with specific data sets. In some aspects, the initial LLM model may be a context-specific LLM or a generic LLM. For example, a generic LLM may be a good starting point for developing a financial LLM by fine tuning with financial context. In some aspects, a financial LLM may be a good starting point for developing a cross -field financial legal LLM. This is because a financial LLM has already been trained on a dataset of financial text and code, so it will have a good understanding of financial concepts. Once the financial LLM has been fine-tuned with legal content, it will be able to perform a variety of tasks, including generating financial and legal documents, such as contracts, agreements, and reports, answering financial and legal questions in a comprehensive and informative way, conducting financial and legal research, and / or identifying and analyzing financial and legal trends. This helps LLM-QA to better understand the context of questions and to generate more relevant answers. In some aspects, the fifth system may further relate to or include a framework (e.g., LangChain or an LLM orchestration framework) for creating Generative Al appliances that enable users to ask appliance operational and business questions. The framework, in some aspects, uses UI / UX content-driven in-context and knowledge graph-refined content to improve the performance of Generative Al models. It provides features such as embeddings, model selection, fine-tuning, and deployment.

[0045] In some aspects, the Generative Al may include a sixth system related to reinforcement learning (RL) 180. RL may be used to improve the performance of LLMs, in some aspects, by providing feedback to the T.T.M 170 on how to generate text that is aligned with specific guidelines or constraints from the context of operational documents or business processing documents. The feedback can be provided in the form of rewards (or penalties) . The proposed approach has three options, (1) take user input, LLM output, and feedback as input data to produce an updated LLM model, (2) take context-enriched user prompt, LLM output,and feedback as input data to produce an updated LLM model, or (3) take user input, context- enriched user prompt, LLM output, and feedback as input data to produce an updated LLM model.

[0046] An autonomous agent, in some aspects, may be a program that can function with or without human intervention. It can make decisions based on information and interact with humans. When working with a human in the loop, it has the capability to take actions according to its understanding of the system. Agent prompt 121 (with human) directs the Al's behavior and output, (while the user prompt initiates the interaction by conveying the user’s intent or inquiry to the AT). Agent prompts 121 are integral to facilitating meaningfill and context-aware interactions between humans and Al systems.

[0047] The Industrial Generative Al Appliance illustrated in diagram 100 presents an Al companion solution, seamlessly adapting to dynamic contexts in accordance with some aspects of the disclosure. With its context switchable feature, businesses gain the ability to effortlessly shift between operational scenarios, optimizing overall efficiency. Supported by an integrated in-context UI / UX Appshell 112 to Industrial Applications, the Industrial Generative Al App fiance, in some aspects, may enhance the user experience fluidity by being tailored to specific tasks.

[0048] Designed as a plug-and-play platform, this system streamlines integration into existing workflows by fine-tuning and eliminating unnecessary complexities. It offers adaptive and efficient Al utilization, enabling users to tap into the potential of existing knowledge repositories. Beyond efficient document processing, it extracts actionable insights, ensuring an unparalleled level of productivity and user satisfaction.

[0049] The first application and application context extraction subsystem, in some aspects, may include interactive application analysis and knowledge extraction. Interactive application analysis and knowledge extraction, in some aspects, may refer to an approach tailored to the modem landscape of UI / UX applications. It may intelligently extract and organize information from these applications, offering valuable insights into UI flows (navigations between particular screens or user interfaces), user interactions, and underlying business actions. The system's seamless integration into diverse application environments makes it an indispensable asset for optimizing user experience and decision-making processes.

[0050] FIG. 2 is a flow diagram 200 illustrating a method associated with the interactive application analysis and knowledge extraction performed by a Generative Al system (e.g., the Generative Al system of FIG. 1) in accordance with some aspects. While illustrated as a sequential set of operations for clarity and ease of explanation, the operations and / or steps illustrated in flow diagram 200 may be performed in parallel or in other orders in accordance with some aspects of the disclosure. The method may begin, at 210 with content extraction. Content extraction at 210 may include employing advanced techniques to extract relevant content from UI / UX applications. The advanced techniques may be used to identify text, images, and other multimedia elements to capture the elements of each screen and interaction (e.g., for determining a meaning and significance of each element captured).

[0051] Based on the content extracted at 210, the method may include (and the system may perform) a UI flow mapping at 220. The UI flow mapping may include an analysis of user pathways to map out the UI flows. For example, the UI flow mapping may track the sequence of screens (or user interfaces), transitions, and user interactions (e.g., selections and / or entries made via the user interface^)) to provide a comprehensive understanding of the application's navigation patterns.

[0052] The method may include, at 230, a business action recognition operation. The business action recognition operation at 230, in some aspects, may include correlating UI interactions with underlying business logic to discern actionable steps taken by users. In some aspects, the business action recognition operation at 230 identifies processes like form submissions, data manipulations, and transactional events that may shed light on the application’s functional dynamics.

[0053] In some aspects, the method may include, at 240, in-context tree creation. The incontext tree creation, in some aspects, may be based on one or more of the content extraction at 210, the UI flow mapping at 220, and / or the business action recognition operation at 230. The in-context tree creation, in some aspects, may be associated with assembling an in-context tree that organizes UI components and interactions hierarchically (e.g., as described in relation to FIGs. 3-6). The in-context tree, in some aspects, may reflect the relationships between screens, features, and / or functionalities, offering a structured overview of the application's architecture.

[0054] The method may include, at 250, a knowledge graph construction operation. Knowledge graph construction operation, in some aspects, may leverage extracted content (e.g., from the content extraction at 210), UI flows (e.g., UI flows identified by the UI flow mapping at 220), and business actions (e.g., identified by the business action recognition operation at 230), the system constructs a dynamic knowledge graph. This dynamic knowledge graph, in some aspects, interconnects data points, interactions, and / or contextual information, fostering a deeper understanding of the application's intricacies.

[0055] In some aspects, one or more knowledge graphs may be a part of self-augmented content for an LLM orchestration framework to obtain the most relevant information for prompt augmentation. For example, an LLM orchestration framework, in some aspects, may use knowledge giaphs to automatically find and add relevant information to prompts, even if the user does not know the relevant content or the location of the content (document). If an administrator prompts the LLM orchestration framework with an inquiry about an error, e.g., a “Tenant creation error,” the LLM orchestration framework may use its knowledge graph to find and add relevant information about a most up-to-date issue log or release log, user manual, and infrastructure eco system that may be used in the tenant creation. This can help the LLM orchestration framework to generate a more informative and comprehensive responses to user.

[0056] Knowledge giaphs, in some aspects, may also be used to make the LLM orchestration framework more user-friendly. If a user does not know the exact name of a piece of content that they want to include in a prompt, the LLM orchestration framework can use its knowledge graph to suggest relevant content. This can help users to save time and effort when creating prompts.

[0057] Additionally, the method may be associated with contextual analysis at 260. The contextual analysis, in some aspects, may identify contextual cues such as user roles, preferences, and usage patterns. In some aspects, the contextual analysis may be used to refine the in-context tree and / or the (dynamic) knowledge graph, enabling a tailored perspective for different user groups.

[0058] FIG. 3 is a diagram 300 illustrating operations associated with generating the (dynamic) knowledge graph and / or in-context tree associated with an application 310 in accordance with some aspects of the disclosure. In some aspects, information is extracted to generate a screen inventory, descriptions for screens and entities, navigation flows, annotationsand interactions, user pathways, and interconnections between screens. An approach that utilizes the extracted information, including screen inventory 351, screen and entity descriptions 352, navigation flows 353, annotations and interactions 354, user pathways (e.g., user flows 355), and interconnections between screens (e.g., screen relationships 356), to create an in-context tree and knowledge graph:

[0059] For example, a set of data extraction and preparation tools (or modules) may include a set of content extraction tools 320 (e.g., a set of programs for performing specific types of content extraction). The set of content extraction tools 320, in some aspects, may include a set of UI exploration tools 321 (e.g., browser-based tools for automating navigation through the screens and / or user interfaces of an application’s UI), a set of behavior analytics tools 322 (e.g., tools for capturing and / or analyzing the behavior of a user of the application’s UI), and / or other content extraction tools. The set of content extraction tools 320, in some aspects, may gather the relevant content, including screen designs, manuals, and / or process documents. In some aspects, a process document may be a document that describes a business process, operation process, or other process in detail. For example, a business process document might describe a process for onboarding new customers, processing orders, or providing customer support, an operation process document might describe a process for manufacturing a product, assembling a product, or delivering a service, and other process documents might describe a process for managing a project, developing a new product, or responding to a crisis.

[0060] The set of content extraction tools 320 may process the gathered content (e.g., the images, documents, etc.) to extract content and / or convert them into a digital format for analysis. In some aspects, the processing may include analyze and extraction relevant information such as extracting the key processes from extracted content, extracting the steps of a process from extracted content, extracting the roles and responsibilities associated with a process, extracting the inputs and outputs of a process from extracted content, and / or extracting the metrics and measures associated with a process from extracted content. For example, the set of content extraction tools 320 may produce application screenshots parameters 331 and / or application screenshot sequences 332. Images and / or text may be provided to a set of text processing tools 340. The text processing tools 340, in some aspects, may include a set of transformer models such as image recognition models 341, natural language processing models 342, and / or machine translation models 343. In some aspects, the text extraction tools (e.g., inone or more of the set of content extraction tools 320 and / or the text processing tools 340) to convert text-based content from images and documents into machine-readable text.

[0061] In some aspects, the set of data extraction and preparation tools may categorize extracted information into groups, such as screen descriptions, entity details, user interactions, etc. The extracted data, in some aspects, may go through a keyword tagging operation to identify and tag keywords, concepts, and entities within the extracted content. In some aspects, this tagging may aid in establishing connections and relationships later.

[0062] For example, in relation to screen inventory 351, the set of data extraction and preparation tools may organize screens into an inventory and / or group them based on their primary functionalities or modules. The set of data extraction and preparation tools may analyze user pathways (e.g., navigation flows 355) by tracing the sequences of screens users typically follow to accomplish tasks. The screen inventory 351 and the navigation flows 355, in some aspects, may form the basis for the hierarchical structure of the in-context tree. The extracted information, in some aspects, may be used to generate mapped data 360 (e.g., in a java script object notation (ISON) or other document format) that may be converted, at 370, into a natural language description associated with the extracted data (e.g., the screen inventory 351 and the navigation flows 355). In some aspects, the system may, at 380, construct the incontext tree 385 by arranging screens in a hierarchical manner, with parent and child relationships representing user navigation. For example, the system may use a library of transformers to generate a sequence of tokens that represent the nodes in the tree and the tokens will be ordered in such a way that they represent the hierarchical relationships between the nodes.

[0063] In some aspects, to generate the knowledge graph 395, the system (e.g., viaalogical grouping function 390) may perform entity extraction and linking to identify entities (objects, concepts, actions) within screen descriptions and annotations and establish links between these entities based on their relationships. In some aspects, identifying the entities may include identifying the entities in the in-context tree (e.g., using LangChain or an LLM orchestration framework), where “entities” may be the nodes, things, and / or elements that are represented in the in-context tree. Additionally, in some aspects, identifying the relationships between the entities based on the connections (or “relationships”) between the entities. In some aspects, the knowledge graph 395 may be a data structure that stores the entities and relationships in the in-context tree. The knowledge graph, in some aspects, may be stored in a variety of formats, such as JSON or resource description framework (RDF).

[0064] For example, the system may utilize the navigation flows and user pathways to define relationships between screens such that screens that are frequently navigated between or share common functionalities are connected in the knowledge graph. The system may further perform concept mapping to map concepts and keywords to relevant screens and entities to establish and / or identify contextual relationships in the knowledge graph.

[0065] In some aspects, the system may further perform enrichment and continuous learning. In some aspects, the enrichment and / or continuous learning may include semantic analysis. The semantic analysis may utilize NLP techniques to perform semantic analysis on the extracted content. This semantic analysis may identify hidden patterns, dependencies, and insights that contribute to the knowledge graph’s enrichment. In some aspects, the system may incorporate user feedback and interactions to refine the relationships in the knowledge graph. Users’ actual behaviors can provide valuable insights into how different screens and entities are related.

[0066] FIG. 4 is a diagram 400 illustrating the extraction of content from a UI / UX associated with an application in accordance with some aspects of the disclosure. Diagram 400 includes a first screen 410 and a set of content and / or context data 412 that may be extracted in some aspects. Diagram 400 also illustrates a transition 420 to a second screen 430 based on the selection of a first element 411 of the first screen 410. A second set of content and / or context data 432 may be extracted, in some aspects, from the second screen 430. The content and / or context extracted from screens 410 and 430 (among others not shown) may enable the generation of an in-context tree related to the UIs (e.g., a website) or to a particular navigation flow (e.g., a checkout process).

[0067] FIG. 5 is a diagram 500 illustrating the elements of a knowledge graph based on the in-context tree of FIG. 4 in accordance with some aspects of the disclosure. Using the example of in-context tree 440 of FIG. 4, the system may use an in-context tree 540 to generate a knowledge graph 550. The knowledge graph 550, in some aspects, may include information relating to a plurality of nodes, edges, and annotations. For example, the knowledge graph may identify nodes 551 that, in some aspects, represent elements or entities identified from a particular screen. The identified edges 553, in some aspects, may represent connectionsbetween nodes (e.g., screens and / or screen elements) identified in the set of navigation flows (e.g., UI / UX flows). The annotations 555, in some aspects, may be natural language descriptions of the node or edge (e.g., attributes and / or functions associated with the node or edge).

[0068] In some aspects, knowledge graphs may be apart of self-augmented content for an LLM orchestration framework to obtain the most relevant information for prompt augmentation. This means that an LLM orchestration framework can use knowledge graphs to automatically find and add relevant information to prompts, even if the user does not know the relevant content or the location of the content (document). For example, if an administrator prompts an LLM orchestration framework to inquire about a “Tenant creation error,” an LLM orchestration framework can use its knowledge graph to find and add relevant information about most updated issue log or release log, user manual, and / or infrastructure eco system used in the tenant creation. This can help an LLM orchestration framework to generate a more informative and comprehensive responses to user. In some aspects, knowledge graphs may also be used to make an LLM orchestration framework more user-friendly. For example, if a user does not know the exact name of a piece of content that they want to include in a prompt, an LLM orchestration framework may use its knowledge graph to suggest relevant content. This can help users to save time and effort when creating prompts.

[0069] FIG. 6 is a diagram 600 illustrating an additional example of content extraction from a screenshot in accordance with some aspects of the disclosure. For example, a screenshot 610 maybe taken from a UI / UX. The system, in some aspects, may analyze the screenshot 610 to extract content and / or content 612 (e.g., via the transformers, or text processing tools 340, of FIG. 3).

[0070] FIG. 7 is a diagram 700 illustrating aspects of content and / or context extraction and analysis in accordance with some aspects of the disclosure. In some aspects, the system uses an approach that utilizes the extracted information, including in-context content 711, operational context in the source system 712, business context in the source system 713, event documents for the operational system 714, event documents for the business system 715, and / or real-time context in the source system 716.

[0071] In some aspects, the in-context content 711 may relate to gathered information that is contextually relevant to the specific tasks and processes within the application. This includesscreen descriptions related documents, user interactions related business processes, and annotations. The operational context in the source system 712, in some aspects, may relate to collected data that provides insights into the operational environment of the application. This could include system configurations, user roles, and operational workflows. In some aspects, business context in the source system 713 may relate to captured information about the application's alignment with type of businesses, business goals, strategies, and objectives. This context might involve understanding how the application supports specific business processes.

[0072] In some aspects, the system may also integrate event documents. The event documents for operational system 714, in some aspects, may relate to integrated event logs or documents from the operational side of the system. These documents, in some aspects, may detail system events, user actions, and / or user interactions, providing a real-time (or near-real- time) operational perspective. In some aspects, the event documents for business system 715 may relate to integrated event logs or documents from the business side of the system. These documents may capture high-level business events, transactions, and processes.

[0073] The system, in some aspects, may further process real-time contextual insights. For example, the system may receive and process real-time context in source system 716 that, in some aspects, may be based on leveraging real-time data feeds to gain insights into the application's current operational state. This includes data on user activities, system performance, and other real-time metrics.

[0074] The aspects of content and / or context extraction and analysis illustrated in diagram 700 may include aspects related to the in-context content tree and the knowledge graph as described above in relation to FIGs. 2-6. For example, the system may integrate the various contextual components (e.g., the in-context content 711, operational context in the source system 712, business context in the source system 713, event documents for operational system 714, and / or event documents for business system 715) into a unified system. The system may construct a knowledge graph that connects the different contextual elements. This graph, in some aspects, may represent relationships between screens, events, user actions, and operational and business contexts. In addition to the knowledge graph, the system may generate (and provide) an in-context representation (e.g., a UI / UX mapping) that visually displays the relationships and connections within the knowledge graph. This representation provides users with an intuitive understanding of how different components relate to each other.

[0075] The system, in some aspects, may perform real-time (or batched) learning and / or optimization operations. For example, the system may perform semantic analysis to provide insights. The semantic analysis techniques, in some aspects, may uncover patterns, trends, and dependencies within the integrated data. This analysis, in some aspects, may enrich the knowledge graph with deeper insights. The real-time learning and / or optimization, in some aspects, may incorporate user feedback and interactions to refine the relationships and contextual connections in the knowledge graph. This iterative process ensures that the representation accurately reflects user needs and behaviors (e.g., as they change over time).

[0076] The system, in some aspects, may perform content tagging 721 (e.g., identifying the entities and relationships in a document and then assigning tags to the entities based on their meaning and context) and content grouping 722 (e.g. , clustering documents together based on the entities and relationships that they share). The additional information related to the incontext tree 731 and knowledge graph 732 provided by the content tagging 721 and the content grouping 722, in some aspects, may be used to support other tasks including: questions and answers generation, entity extraction, text summarization, machine translation, and / or natural language generation.

[0077] This approach may provide the following benefits: holistic understanding, informed decision-making, real-time insights, and / or an optimized user experience. For example, the approach may integrate various types of contextual information, providing users with a holistic understanding of the application's operations and alignment with business goals. In some aspects, the knowledge graph and in-context representation may enable stakeholders to make informed decisions based on the interplay between technical and business contexts. By incorporating real-time context and event data, the approach, in some aspects, may keep users updated with current operational situations. Additionally, in some aspects, in-context content combined with real-time and operational contexts ensures that risers receive tailored and relevant information.

[0078] FIG. 8 illustrates a set of components of the system associated with training the system in accordance with some aspects of the disclosure. Elements of FIG. 8, in some aspects, may correspond to similarly numbered and / or labeled elements of FIG. 1. To improve the performance of an ELM (e.g., LEM 870) on a question answering task, a knowledge gr aph (e.g., operational knowledge graph 851 or business knowledge graph 861) may be used to extract subjects, relationships, and objects (e.g., content in scope 853 and / or content in scope863) from a dataset of documents (e.g., operational documents 852 and / or business documents 862) that contain text about these entities. These extracted entities (e.g., subjects, relationships, and objects) can then be used to create questions and answers (e.g., via questions and answers generator 855 and / or questions and answers generator 865), which can be mapped to predefined embeddings (e.g., common embeddings 856, common embeddings 866, common embeddings 868). Finally, the LLM 870 (e.g., the operational LLM 871 and / or the business LLM 872) may be fine-tuned (e.g., via fine-tuning 857 and / or fine-tuning 867) on a dataset of labeled data. This will help the LLM to leam the relationships between words more easily and improve its performance on the question answering task.

[0079] The steps associated with the training described above, in some aspects, may include extracting subjects, relationships, and objects from documents (e.g., operational documents 852 and / or business documents 862). This step, in some aspects, may involve using a knowledge graph to identify the subjects, relationships, and objects in the documents. For example, if there is a document about the entity "John Smith," the knowledge graph may tell us that John Smith is a person, that he is married to Jane Doe, and that he has two children. The training, in some aspects, may further include creating questions and answers from the extracted subjects, relationships, and objects. For example, the training may create the question "Who is John Smith married to?" and the answer "Jane Doe."

[0080] The training may, in some aspects, include mapping questions and answers to predefined embeddings. In some aspects, embeddings may be a way of representing words as vectors of numbers. This allows the LLM to leam the relationships between words more easily. In some aspects, the training may include fine-tuning the LLM on a dataset of labeled data. This is done by adjusting the weights of the LLM to improve its performance on the question answering task.

[0081] This approach improves the performance of an LLM on a question answering task by helping it to leam the relationships between words more easily. While LLMs may be used to perform a variety of tasks, to perform these tasks effectively, LLMs may benefit from training on relevant content.

[0082] In some aspects, there may be two core types of content that can be used to train an LLM: documents and know-how. Documents, in some aspects, may be anything from product specifications to customer support tickets . The documents may provide the LLM with a wealthof information about the domain in which it will be used. Know-how, in some aspects, may be the tacit knowledge that experts in a particular domain possess. It can be difficult to capture and codify know-how, but it may be leveraged to allow the LLM to be able to perform tasks that require human-level understanding. For example, training an LLM to perform tasks related to an application, may use the following types of content: application operation status content or business content that is associated with the application. The application operation status content, in some aspects, may include information about the current state of the application, such as the number of active users, the number of errors that have been reported, and the performance of the application’s various components. The business content, in some aspects, may include information about the business processes that are supported by the application, such as order fulfillment, customer support, and inventory management. By training the LLM with both types of content, it may be able to better understand the domain in which it will be used and perform tasks more effectively.

[0083] FIG. 9 is a diagram 900 illustrating components of the system associated with an inferencing operation in accordance with some aspects of the disclosure. In some aspects, the inferencing operation may begin with the entry of a query (e.g., associated with one of Agent prompts 921 or user prompts 922) into the application (e.g., by an agent or a user). In some aspects, an AppShell 912 may extract the context of the query from the application’s data. A an LLM orchestration framework(e.g., associated with one or more of user prompts 922, common embeddings 932, a context enriched user prompt 933, a content and document extraction unit 941, a set of sub-topic documents 942, common embeddings 943, and / or vector store 944) may then use the context to extract the key entities and relationships from the query. The embedding (e.g., the common embedding 932 or the common embedding 943) and vector database (e.g., vector store 944) may be queried to find the most similar queries to the current query. Once the query has been processed to identify the elements (e.g., content and context) of the query, the LLM 970 (e.g., one of the operational LLM 971 or the business LLM 972 based on the context identified for the query) may be used to generate a response to the query and the response may be provided (output or displayed) to a user associated with the query. For a query generated by an agent the response may be a warning or notification to a user identified as being associated with the query (where the inferencing operation may include identifying one or more users associated with a relevant role or status).

[0084] An agent, in some aspects, may be a software program that is responsible for monitoring a system and taking action in response to system events. Agents may be used to monitor a variety of systems, including hardware, software, and networks. Generally, agents have two primary functions: monitoring and event triggering / response. For example, in some aspects, agents may continuously collect data from the system they are monitoring. This data can include information about the system's health, performance, and utilization. Additionally, agents, in some aspects, may be configured to trigger events based on certain conditions. For example, an agent could be configured to trigger an event if the system’s CPU usage exceeds a certain threshold. When an event is triggered, the agent can take action to respond to the event. This action could involve sending an alert, taking corrective action, or logging the event.

[0085] In addition to the elements illustrated in FIG. 1, diagram 900 includes an agent prompt interface 923 and / or a set of agent prompts 921 that, in some aspects, may allow a user to interact with one or more agents. Agent prompt interface 923, in some aspects, may allow one or more users to view the status of the system, view the events that have been triggered, and view the agent’s inquires. In some aspects, the agent's behavior may be configured through policies . For example, a user could use the interface to receive agent ’ s information of the status of the system's CPU usage. The user could also use the prompt interface to view the events that have been monitored and triggered by the agent. If the user sees that the CPU usage is too high, they could use the prompt interface to configure the agent to take corrective action, such as sending an alert or throttling the system’s performance. Accordingly, agents and agent prompt interfaces, in some aspects, may be used to monitor and manage a variety of systems. They can be a valuable tool for ensuring the health and performance of critical systems.

[0086] Diagram 900 further illustrates a neural network (NN) 990 (as an example of a machine-trained network) that may be associated with one of the context-specific LLMs (e.g., one of the operational LLM 971 or the business LLM 972). The NN 990, in some aspects, may have an input layer 991 representing the input to the NN 990, one or more hidden layers 992, and an output layer 993 including one or more output values. Each set of values in any layer may be based on, or associated with, a set of weights between layers (e.g., weights reflecting a relationship between (or the contribution of) each node in one or more previous layers to each node in a particular layer). During the training process of an LLM, in some aspects, the set of weights of the neural network may be adjusted to minimize the error between the predicted and actual words. The set of weights associated with the trained NN 990, in some aspects, encodeand / or represent the relationship between words (or other units used to define the nodes of a neural network).

[0087] In some aspects, a plurality of NNs 990 may be trained based on different models. For example, a first NN 990 may be trained based on particular words (e.g., based on a continuous bag of words (CBOW) model), a second NN 990 may be trained based on word positions (e.g., based on a skip-gram algorithm / model), and a third NN 990 may be trained based on context (e.g., based on nodes in a knowledge graph and a skip-gram algorithm / model). The plurality of NNs 990, in some aspects, may be trained for each of a plurality of contexts and / or personas. Based on the separately trained NNs 990, the system may generate one or more common embeddings (e.g., common embeddings 932 and 943). The one or more common embeddings (e.g., common embeddings 932 and 943), in some aspects, may be associated with a hidden layer of a corresponding NN 990 that may be configured to combine the information regarding relationships between words and / or nodes represented in the plurality of NNS 990 separately trained to identify different types of relationships. FIGs. 10, 11, and 12 below illustrate different ways that a common embedding may be generated, produced, or constructed based on the intended use.

[0088] In some aspects, an NN 990 may be trained for a specific business context. A business context in the context of training an LLM, in some aspects, may refer to a specific industry or domain that the LLM will be used for. The context may indicate, or be associated with, factors such as the types of tasks that the LLM will be expected to perform, the types of data that the LLM will be exposed to, and the specific requirements of the business. For example, an LLM that is being trained to provide customer support in the financial services industry may be configured to account for (or be famihar with) the specific products and services that the company offers, as well as the regulatory environment in which the company operates. Whereas, an LLM that is being trained to generate marketing copy for a technology may be configured to account for (or be famihar with) the company's products and services, as well as the target audience for the marketing copy.

[0089] In some aspects, an NN 990 may be trained for a specific operational context. An operational context in the context of training an LLM, in some aspects, may refer to the specific ways in which the LLM will be used and fine-tuned. For example, the operational context to fine tune an LLM may be associated with automating the classification and triage of errors, such that LLMs can be trained with operational context to identify and classify different typesof errors, as well as to prioritize errors based on their severity and impact. This can help to reduce the time it takes to respond to errors and to ensure that the most critical errors are addressed first. In some aspects, the operational context to fine tune an LLM may be associated with generating root cause analysis such that the LLMs can be trained to analyze error logs and other data to identify the root cause of errors from system operational context. This can help to prevent errors from recurring and to improve the overall reliability of technology systems. The operational context to fine tune an LLM, in some aspects, may be associated with generating corrective actions such that the LLMs can be trained to generate corrective actions for different types of errors from operational context. This can help to reduce the time it takes to resolve errors and to improve the overall efficiency of error response.

[0090] Similarly, in some aspects, an NN 990 may be trained for a specific persona for a business or operation context. In some aspects, a persona for a business context may represent the user (or customer) of a product or service, from a business perspective. For example, persona-specific fine-tuning of an LLM refers to the process of training an LLM to perform specific tasks or to generate content that is tailored to a specific persona. This can be achieved by training the LLM on a dataset of text and code that is representative of the desired persona, in production to reduce downtime, anomaly, facilitating system failure mode triages, etc. This persona-specific fine-tuning, in some aspects, may include factors such as the types of systems and / or infrastructure that the business context and business processes will be integrated with and / or enabled with, the types of data that the system will be exposed to in production, and the specific performance requirements for the business context in scope. A persona, in some aspects, considers factors such as the customer's needs, goals, and pain points, as well as the business's overall objectives. A persona for an operational context, in some aspects, may represent the user (or customer) of a product or service, from an operational perspective. In some aspects, it may considers factors such as the customer’s workflows, systems, and processes, as well as the operational requirements of the product or service.

[0091] For example, a business persona may include information related to a name of the user, a title a company, an industry, a set of identified goals, and a set of identified pain points (e.g., Name: John Smith; Title: Sales Manager; Company: Acme Corporation; Industry: Technology; Goals: increase sales by 10% year-over-year, and improve customer satisfaction; Pain points: difficulty tracking customer interactions, time-consuming manual data entry, and lack of visibility into sales pipeline. Similarly, an operational persona may include informationrelated to a name of the user, a title a company, an industry, a set of identified goals, and a set of identified pain points (e.g., Name: Jane Doe; Title: CRM Administrator; Company: Acme Corporation; Industry: Technology; Goals: automate CRM workflows, improve data quality, integrate CRM system with other systems; Pain points: downtime triage, lack of IT resources, difficulty maintaining CRM system, lack of training for users.)

[0092] FIG. 10 is a diagram 1000 illustrating a first embedding technique in accordance with some aspects of the disclosure. In some aspects, a superposition technique may be used. The superposition technique, in some aspects, may involve adding weights associated with word embeddings (e.g., a continuous bag of words model), position embeddings (e.g., associated with a skip-gram model), and node embeddings (e.g., associated with a skip-gram model) together to create a new embedding. Word embeddings, in some aspects, represent the meaning of words, position embeddings represent the position of words in a sentence, and node embeddings represent the meaning of entities in a knowledge graph. By adding these embeddings together, the system may create a more comprehensive representation of the input text. In some aspects, the addition of the embeddings includes adding the weights associated with one or more hidden layers from neural networks trained for each of a word embedding, a position embedding, and anode embedding. The superposition technique, in some aspects, may be a good choice for performing a task that is associated with understanding the meaning of words and their relationships.

[0093] FIG. 11 is a diagram 1100 illustrating a second embedding technique in accordance with some aspects of the disclosure. In some aspects, a dimension extension technique may be used. The dimension extension technique, in some aspects, may involve adding weights associated with word embeddings (e.g., a continuous bag of words model) and position embeddings (e.g., associated with a skip-gram model) and then appending node embeddings (e.g., associated with a skip-gram model) to the resulting embedding. This dimension extension technique, in some aspects, may allow the system to preserve the information from the word and position embeddings, while also adding the information from the node embeddings. In some aspects, the addition of the embeddings includes adding the weights associated with one or more hidden layers from neural networks trained for each of the word embedding and the position embedding. The dimension extension technique, in some aspects, may be a good choice for performing a task that is associated with understanding the meaning of entities and their relationships.

[0094] FIG. 12 is a diagram 1200 illustrating a third embedding technique in accordance with some aspects of the disclosure. Ln some aspects, an embedding population expansion technique may be used. The embedding population expansion technique, in some aspects, may involve creating two embeddings: a first combined embedding that is based on the addition of word embeddings and position embeddings, and a second combined embedding that is based on the addition of node embeddings and position embeddings. This embedding population expansion technique, in some aspects, may allow the system us to represent the input text in two different ways, which can be helpful for certain tasks. In some aspects, the addition of the embeddings includes adding the weights associated with one or more hidden layers from neural networks trained for different embeddings. The embedding population expansion technique, in some aspects, may be a good choice for performing a task that is associated with representing the input text in two different ways.

[0095] FIG. 13 is a diagram 1300 illustrating a high-level view of the operations discussed above in relations to FIGs. 1-12 in accordance with some aspects of the disclosure. Diagram 1300 illustrates that an initial LLM 1310 (e.g., a generic LLM) may be generated by a large corpus of words 1305 (e.g., using a public source such as Wikipedia™). The initial LLM 1310, in some aspects, may go through a fine-tuning model 1320 to be fine-tuned based on a taskspecific dataset 1315 to produce a context-specific (or task-specific) LLM 1330 that may be tested based on test dataset 1325.

[0096] In some aspects, the system may include a question and answer generator (e.g., question and answer generator 155 or 165 of FIG. 1, or questions and answers generator 855 or 865 of FIG. 8). FIG. 14 is a diagram 1400 illustrating components of the system associated with training a question and answer generator in accordance with some aspects of the disclosure. In some aspects, a question and answer (QA) generator is a type of Al model that may be used to create questions and answers from a given dataset. This can be useful for a variety of purposes, such as creating industrial content, generating test questions, or improving the performance of LLMs.

[0097] In some aspects, LLMs are large language models that are trained on massive datasets of text and code. This training allows the LLMs to learn the statistical relationships between words and phrases, which enables them to generate text, translate languages, write different kinds of creative content, and answer your questions in an informative way. However, LLMs are not always able to answer questions accurately or informatively. This is because theyare not always familiar with the specific terminology and concepts that are used in a particular domain. A QA generator, in some aspects, may be used to address this issue by creating questions and answers that are specific to a particular domain. This can help LLMs to leam the specific terminology and concepts that are used in the domain, which can improve their ability to answer questions accurately and informatively.

[0098] The data components (e.g., data used to train the context-specific QA generator), in some aspects, may include client data, converted customer data (e.g., image, text, and / or audio), large diffusion model for images, or other proprietary models. Client data, in some aspects, may be used to fine-tune a pretrained LLM. This data can include customer support tickets, product reviews, and other forms of customer feedback. This helps the model to leam how to interact with customers in a natural and informative way. In some aspects, customer data (e.g., image, text, and / or audio) may be converted into input that can be used to train a pretrained (initial) LLM. This can be done using a variety of techniques, such as NLP and computer vision (CV). The converted customer data may be associated with an operation to convert customer datasets to image to text 1411.

[0099] A large diffusion model for images, in some aspects, may be a type of Al model that can be used to generate realistic images. This model, in some aspects, may be trained on a massive dataset of images, which allows it to leam the statistical relationships between pixels. This enables the model to generate images that are both realistic and visually appealing. The large diffusion model may be associated with generating image / videos using diffusion model 1412. In addition to the models mentioned above, there may be a number of other proprietary models that can be used to generate a Generative Al co-pilot (or QA generator). These models, in some aspects, may be trained on a variety of datasets, including text, code, and images. This allows them to leam the statistical relationships between different types of data, which enables them to generate creative and informative content. The additional models may be associated with the additional models 1413.

[0100] The QA generator training, in some aspects, may be associated with a pretrained LLM 1401 . The pre trained LLM 1401, in some aspects, may be a type of Al model that has been trained on a massive dataset of text and code. This training allows the LLM to leam the statistical relationships between words and phrases, which enables it to generate text, translate languages, write different kinds of creative content, and answer user’s questions in an informative way. Confidential questions and answers 1402, in some aspects, may be createdon-premises using the pretrained LLM 1401. This allows organizations to control the data that is used to train the model and to ensure that the model is not exposed to sensitive information.

[0101] Once a pretrained LLM has been created, it can be fine-tuned to produce a private LLM question generator 1421 and / or a private LLM answer generator 1432 using customer data. This process involves feeding the model a dataset 1422 of questions and answers that are specific to the organization's domain. This helps the model to learn the specific terminology and concepts that are used in the organization. The training, in some aspects, may include a human in the loop to correct 1431. Once a pretrained LLM has been fine-tuned, it may be important to have a human in the loop to correct any errors that the model may make. This helps to ensure that the model is providing accurate and reHable information.

[0102] Once a pretrained LLM has been fine-tuned and corrected (e.g., to produce a private LLM question generator 1421 and / or a private LLM answer generator 1432), in some aspects, it may be deployed in a model serving environment (e.g., a model server 1440). This allows the model to be accessed by users through a variety of channels, such as a web application, a mobile app, or a chatbot.

[0103] In some aspects, the training may involve pretrained large language models 1401 and / or confidential answers — on premises or on cloud 1402. Deployment, in some aspects, may be via an open source LLM like LLaMa. These models are designed to be used in various NLP tasks, such as language translation, text classification, and language generation. In some aspects, these models are based on open-source data (e.g., non-confidential data — on cloud 1403), which means that the data used to train them is available to the public and has been shared under a Hcense that allows for use and modification. This is different from proprietary models, which are trained on private data and are not publicly available. The weights of these models refer to the learned representations of the data used to train them. These representations are the result of complex algorithms that analyze the data and learn patterns and relationships within it. The weights are the values assigned to these patterns and relationships, which are used to make predictions or generate text.

[0104] In some aspects, the training may involve a Dataset for model customization 1422, Customer Dataset 1410 and Convert Customer Dataset to Tasks using ChatGPT or proprietary models 1404. Creating or fine-tuning a dataset for training a LLM (or GPT) model for conversational tasks involves gathering and preparing a dataset of text data that can be used totrain the model. The goal is to create a dataset that can help the model learn to generate coherent and contextually relevant text responses to given prompts or input. In some aspects, the system may define the scope and purpose of the dataset. For example, the system may determine the specific conversational task for the model to perform, such as answering questions, generating product descriptions, or providing customer support using the customer original dataset like pdf, word documents, html pages, images, videos, and / or audio. For example, in relation to diagram 1400, the task is a question answering task.

[0105] In some aspects, the system may collect and preprocess the data. For example, the system may gather text data from various sources, such as books, articles, and websites. A preprocessing of the data may, in some aspects, include cleaning, tokenizing, and normalizing the data to create a consistent format for training. The preprocessed data may be used to create training examples that the model can learn from. In some aspects, the training may involve labeling the training examples by assigning labels or categories to the training examples to help the model understand the context and purpose of the text data. The generated data set, in some aspects, may be divided and / or split into training, validation, and test sets (e.g., the dataset may be divided such that 80% is for training, 10% is for validation and 10% is for testing).

[0106] The function of the human in the loop, or reinforcement learning with human feedback (RLHF), curation, in some aspects, may include having a human review the activity of (or interact with) the system to enhance, determine the accuracy of the relevant responses, and to avoid harmful content from the models. In some aspects, the model server may be a system that stores and serves the different models for different copilots (e.g., Generative Al companions) likePentaho 1451, HCP 1452, or other programs (e.g., represented by APP 1453).

[0107] In some aspects, generating a dataset for the training may involve image to text, generative models for images and videos and additional models. For example, a text-to-images process, in some aspects, may involve a computer model generating an image based on a given text description. This process involves using NLP techniques to analyze the text and generate an image that corresponds to the described content. The general steps involved in the text-to- images process using Al models may include multiple steps. In some aspects, the text-to images processing may involve text preprocessing to remove any irrelevant information, such as punctuation, stop words, and special characters. This helps the model focus on the meaningful content of the text. The preprocessed text, in some aspects, may then be converted into numerical representations using techniques such as word embeddings or character embeddings.These embeddings capture the semantic meaning of the text and are used as input to the Al model. In some aspects, the Al model may then use the text embeddings as input and generates an image or video based on the described content. This image can be a simple image, an illustration, a complex scene, or a video depending on the complexity of the text description. The generated image may not always be accurate or visually appealing. Therefore, the image, in some aspects, may be refined through additional processing steps, such as filtering, resizing, or color adjustments. The text-to images process may end with the final image being output, which can be used for various applications such as image generation, object detection, or scene understanding.

[0108] FIG. 15 is a diagram 1500 illustrating a set of elements relating to the training of a question generator (QG) in accordance with some aspects of the disclosure. In some aspects, open source GPT API 1512 or proprietary GPT-like API may be used to extract examples of conversations and questions to maintain quahty, avoid bias, and prevent harmful content. For example, in some aspects, the training may use publicly available data sources (e.g., such as a public dataset 1511) that have been carefully curated and anonymized to maintain privacy and security. In some aspects, the training may be used to develop Al language models that are transparent, explainable, and inclusive, and that avoid perpetuating biases and stereotypes. In some aspects, the training may involve implementing robust quahty control measures to ensure that the examples generated are accurate, relevant, and free from harmful questions. For example, a user may provide guidelines on how to identify and avoid undesirable, illegal, and / or harmful questions In some aspects, one or more users may be encouraged to continuously monitor and improve the questions, and provide guidelines on how to do so in a responsible and ethical manner.

[0109] FIG. 16 is a diagram 1600 illustrating a set of elements relating to the training of am answer generator (AG) in accordance with some aspects of the disclosure. In some aspects, open source GPT API 1612 or proprietary GPT-like API may be used to extract examples of conversations and answers to maintain quality, avoid bias, and prevent harmful content. For example, in some aspects, the training may use publicly available data sources that have been carefully curated and anonymized to maintain privacy and security. In some aspects, the training may be used to develop Al language models that are transparent, explainable, and inclusive, and that avoid perpetuating biases and stereotypes. In some aspects, the training may involve implementing robust quahty control measures to ensure that the examples generatedare accurate, relevant, and free from harmful questions. For example, a user may provide guidelines on how to identify and avoid undesirable, illegal, and / or harmful answers. In some aspects, one or more users may be encouraged to continuously monitor and improve the questions, and provide guidelines on how to do so in a responsible and ethical manner.

[0110] In some aspects, the elements illustrated in diagrams 1500 and 1600 may be associated with fine tuning open source LLMs (e.g., public API 1512, original foundation model 1521, public API 1612, original foundation model 162L LLaMav2, and / or Open Alpaca foundational models.) For example, the system may fine tune dataset creation based on human creation or GPT-like creation on premise (e.g., privately or proprietarily). In some aspects, RLHF 1513 and / or RLHF 1613 may be used to mitigate out of domain answers and hallucinations (non-sensical answers). In some aspects, the system may be associated with an enhanced LLM for amplifying the spectrum of use cases beyond Q&A like Text2Images, GenAI for Images, Sentiment Analysis, Entity Recognition, Conversational, and / or Summarization. Multiple Q&A generators, in some aspects, may be trained for corresponding multiple products using a same foundational model by tailoring and / or fine-tuning an LLM with customized datasets. The different Q&A generators, in some aspects, may be associated with different contexts (e.g., business or operational contexts, or persona-based contexts including business / operational context as well as characteristics of the user).

[0111] In some aspects, the elements illustrated in diagrams 1500 and 1600 may be associated with automation of the question and answer process for generation of a fine tuned dataset. The process may use a first LLM to generate questions for fine tuning a second LLM and may create a downstream task of the LLM generation of the questions. This LLM, in some aspects, may be an “expert” in generating questions about a topic by using RL for optimizing the response of those questions. The RL may use rewards to fine tune the Question LLM model and then deploy it to generate questions based on context. In some aspects, another LLM may be the answers expert model. The process, in some aspects, may include using the other LLM to generate answers for fine tuning an LLM model and create a downstream task of the LLM generation of the answers. This LLM may be an “expert” in generating answers about the topic because we can use RL for optimizing the response of those questions. The RL may use the rewards to fine tune the Answer LLM model and then deploy it to generate answers based on questions and context.

[0112] FIG. 17 is a diagram 1700 illustrating a method associated with the use of a framework (e.g., LangChain or an LLM orchestration framework) associated with the system in accordance with some aspects of the disclosure. In some aspects, the system may be associated with an LLM orchestration framework in the Generative Al Appliance for industrial usage that uses the content of the functions of document ingestion (data loading, document splitting, etc.), content embedding generation, vector database integration, prompt engineering, large language model integration, model output evaluation, and prediction output formatting.

[0113] At 1710, the system may perform document ingestion. The document ingestion in the an LLM orchestration framework process, in some aspects, may be performed to ingest the documents and context from the application and application documents used to train the model. This can be done by loading the documents into a data lake or database. The documents can then be split into smaller units, such as sentences or paragraphs, to make them easier to process.

[0114] At 1720, the system may perform a content embedding generation and / or mapping. Once the documents have been ingested at 1710, the next step, in some aspects, may be to generate content embeddings. Content embeddings in some aspects, may be numerical representations of the content of the documents (e.g., within the weights of a NN). They are typically created using MLP techniques.

[0115] At 1730, the system may perform a vector database integration. The content embeddings generated at 1720 may then be integrated into a vector database at 1730. A vector database, in some aspects, may be a database that is optimized for storing and retrieving vector data. This allows the content embeddings to be quickly and efficiently accessed by an LLM model later.

[0116] At 1740, the system may perform prompt engineering. The next step, in some aspects, may be to engineer prompts. Prompts are the instructions that are given to the LLM orchestration pipeline. They tell the model what to do with the content embeddings. Prompts can be generated manually or automatically. Ln this case, prompts may be enriched by the content from vector database.

[0117] At 1750, the system may perform an LLM integration. A selected LLM, in some aspects, may then be integrated with existing LLM orchestration pipeline. This allows the LLM to generate text that is both coherent and informative.

[0118] At 1760, the system may perform a model output evaluation: The output of the LLM, in some aspects, may then be evaluated at 1760. This is done to ensure that the model is generating text that is accurate, relevant, and creative. The output of the model may be evaluated manually or automatically.

[0119] At 1770, the system may perform a prediction output formatting. The system may complete this component of the inferencing (e.g., the component related to the LLM) by formatting the prediction output. This may include, in some aspects, converting the text to a desired format, such as HTML or PDF. The prediction output may then be presented to the user.

[0120] FIG. 18 is an illustration of a set of operations associated with a conversational UI to generate an answer for a question and / or prompt in accordance with some aspects of the disclosure. A conversational UI 1810 may be associated with a file uploader 1812. The conversational UI 1810, in some aspects, may receive a question prompt 1820 associated with a set of files provided to file uploader 1812. The files may be divided into data chunks 1814 and the question prompt 1820 and the data chunks 1814 may be processed by the embedding model 1830 to produce a data embedding 1831 and a query embedding 1832 that may be used to identify the content and context associated with the question prompt 1820. For example, a vector database 1840 may, based on the data embedding 1831 and the query embedding 1832, identify (e.g., via an API gateway 1841 and a load balancer 1842) a particular partition and / or shard (e.g., associated with data 1845, data 1846, or data 1847) of the partitions / shards 1844 using a query engine.

[0121] The vector database 1840, in some aspects, may produce a prompt 1860 that may be processed through a context-specific LLM 1870 (identified based on the context identified via the vector database 1840). The context-specific LLM 1870 may, based on the prompt 1860, generate an answer 1875 that may be formatted for outputting via a conversational UI output panel 1895.

[0122] FIG. 19 is a diagram 1900 illustrating a set of elements of a reinforcement learning associated with the system in accordance with some aspects of the disclosure. A reinforcement learner 1980 (e.g., a reinforcement learning component), in some aspects, may be used to update the large language model using the user feedback or automated algorithmic feedback. The algorithmic feedback may come from another independent machine learning (or machine -trained) model. For example, a sentiment predictor model which would be trained for predicting the user sentiment based on an input sentence. The reinforcement learner 1980, in some aspects, may be trained using multiple data options depending on desired user experience. A first option for training may take user input, LLM output, Human or algorithmic feedback as input data and produce an updated LLM model. In some aspects, a second option for training may take a context-enriched user prompt, LLM output, Human or algorithmic feedback as input data and produce an updated LLM model. A third option for training may take context enriched user prompt, LLM output, Human or algorithmic feedback as input data and produce updated LLM model.

[0123] The context enriched user prompt 1930 maybe generated using one, some, or all of the embeddings (e.g., word embedding 1921, position embedding 1922, or node embedding 1923) depending on the specific design for customer use case in the Generative Al Appliance. In some aspects, reinforcement learning may be a type of machine learning that allows an agent to learn how to behave in an environment by trial and error. The reinforcement learning agent is given a reward for taking actions that lead to desired outcomes, and a penalty for taking actions that lead to undesired outcomes. Over time, the agent learns to take actions that maximize its reward.

[0124] As discussed above, a Generative Al appliance, in some aspects, may be a system that uses artificial intelligence to generate text, translate languages, write different kinds of creative content, and answer questions in an informative way. The appliance typically consists of an LLM, a knowledge graph, and an NLP engine. In some aspects, for the reinforcement learning and its interactions, after user input, the process may be associated with: embedding (e.g., in association with one or more of word embedding 1921, position embedding 1922, or node embedding 1923), an enriched user prompt 1930, an LLM 1940, LLM output 1950, human / agent in the loop feedback 1960, a reinforcement data processor 1970, a reinforcement learner 1980, and / or an LLM 1990 updates.

[0125] In some aspects, as discussed above, an embedding may be a process of representing text or other data as a vector of numbers. The embeddings, in some aspects, may include one or more of a word embedding 1921, position embedding 1922, or node embedding 1923 as described in relation to FIGs. 10, 11, and 12, above. This allows the data to be processed by machine learning models. An enriched user prompt 1930, in some aspects, may be a prompt that has been augmented with additional information, such as the user's context or the currentstate of the environment. Uris information can help the LLM 1940 to generate more relevant and informative output. An LLM 1940, in some aspects, may be a type of Al model that has been trained on a massive dataset of text. This training allows the LLM 1940 to learn the statistical relationships between words and phrases, which enables it to generate text, translate languages, write different kinds of creative content, and answer questions in an informative way. The LLM 1940 may produce LLM output 1950 that may include text that is generated by the LLM. This output can be used to answer questions, generate creative content, or translate languages.

[0126] The LLM output 1950 may be provided to a human for human / agent in the loop feedback 1960. The human / agent in the loop feedback 1960 (or algorithmic feedback), in some aspects, may be feedback that is provided by a human or agent to the LLM. This feedback can be used to improve the performance of the LLM by providing it with information about the desired outcome. In some aspects, the reinforcement learner 1980 may be trained using a dataset of enriched user prompts 1930 and LLM outputs 1950. The enriched user prompts 1930, as described above, may be augmented with additional information, such as the user’s context or the current state of the environment. This information can help the reinforcement learner to learn how to generate text that is more relevant and informative to the user's needs. In some aspects, the reinforcement data processor 1970 may be associated with a process that collects and processes data from the environment. This data can be used to train the reinforcement learner 1980. The LLM outputs 1950 connecting to reinforcement data processor 1970, in some aspects, may be used to provide feedback to the reinforcement learner 1980 The feedback can be used to improve the performance of the reinforcement learner 1980 by providing it with information about the desired outcome.

[0127] In some aspects, the reinforcement learner 1980 may be a machine learning model that learns how to take actions in an environment by trial and error. The reinforcement learner 1980 may, for example, be given a reward (e.g., receive positive feedback) for taking actions that lead to desired outcomes, and a penalty for taking actions that lead to undesired outcomes. Over time, the reinforcement learner, in some aspects, may learn to take actions that maximize its reward. The reinforcement learner 1980, in some aspects, may produce LLM updates 1990. In some aspects, the LLM updates 1990 may be (or may indicate) changes that to be made to the LLM 1940. These updates can be made to improve the performance of the LLM by makingit better at generating text, translating languages, writing different kinds of creative content, or answering questions in an informative way.

[0128] FIG. 20 is a flow diagram 2000 illustrating a method of generating output associated with an industrial application in accordance with some aspects of the disclosure. In some aspects, the method is performed by a Generative Al appliance or related apparatus and / or components of an industrial application (e.g., implemented by one or more computer devices such as computer device 2205) that performs various analyses and / or processing operations and generates various recommendation and / or responses. At 2010, the apparatus may extract content and context information associated with the industrial application. In some aspects, the content and context information may include information regarding a set of workflows associated with the industrial application. Each workflow in the set of workflows, in some aspects, may include one or more user interfaces (e.g., UIs) of the industrial application, one or more selections via the one or more user interfaces, one or more entries via the one or more user interfaces, or one or more documents associated with the one or more user interfaces of the industrial application. In some aspects, the content information regarding the particular workflow may include, first content provided by the industrial application via the set of one or more user interfaces, data provided to the industrial application via the set of one or more user interfaces, and / or a set of documents provided to the industrial application via the set of one or more user interfaces. For example, referring to FIGs. 1-12 and 14-19, the Gen Al UI / UX 112 may extract content and / or context information (e.g., at 210 or 1710-1730) associated with an industrial application 110.

[0129] In some aspects, the context information regarding a particular workflow in the set of workflows may include a set of one or more user interfaces that are accessed by a user and a set of selections made via the set of one or more user interfaces. The context information regarding the particular workflow, in some aspects, may further include navigation information regarding a navigation through the set of one or more user interfaces via the set of selections. In some aspects, the context information may be used to generate a tree graph (an in-context tree) indicating relationships between a plurality of user interfaces associated with the industrial application. For example, referring to FIGs. 2-6, the content and / or context information may be used to generate the in-context tree 440 / 540.

[0130] The one or more context-specific LLMs, in some aspects, may be associated with a first embedding associated with a current word, a second embedding associated with a currentword position, and a third embedding associated with a knowledge graph based on the tree graph, and where the one or more context-specific LLMs use one of (1) a first superposition of the first embedding, the second embedding, and the third embedding, (2) the third embedding and a second superposition of the first embedding and the second embedding, or (3) the second superposition and a third superposition of the second embedding and the third embedding. For example, referring to FIGs. 1 , 8-12, and 17-19, the apparatus and / or method may be associated with common embeddings 132 / 932, 143 / 943, 156 / 856, 166 / 866, 168 / 868, the embeddings illustrated in diagrams 1000, 1100, and 1200, or in relation to content embedding generation and / or mapping at 1720.

[0131] At 2020, the method (or apparatus) may identify, produce, or update one or more context-specific LLMs (by adjusting, if producing and / or updating the context-specific LLM) one or more LLMs based on the content and context information. In some aspects, the one or more context-specific LLMs may include at least a first context-specific LLM for a first context and a second context-specific LLM for a second context. The first context, in some aspects, may be associated with a business context and the second context may be an operational context. In some aspects, the one or more context-specific LLMs may include a plurality of user-specific LLMs for a corresponding plurality of users. In some aspects, as part of identifying, producing, or updating one or more context-specific LLMs, the method (or apparatus) may produce a plurahty of user-specific (or persona-specific) LLMs associated with a first context-specific LLM by separately adjusting the first context-specific LLM for each of a corresponding plurahty of users associated with a first context. In some aspects, producing a particular user-specific LLM of the plurality of user-specific LLMs for a particular user in the corresponding plurahty of users may include adjusting the first context-specific LLM based one or more of a role associated with the particular user, a status associated with the particular user, a set of stored actions associated with the particular user, or an application associated with the particular user. For example, referring to FIGs. 1 , 9, and 13-19, the apparatus and / or method may be associated with an initial LLM 1310 or pretrained LLM 1401, pubhc API 1512, public API 1612 with an adjustment associated with one or more of a fine-tuning 157 / 857, 167 / 867, fine-tuning model 1320, any of supervised fine-tuning model 1523 or 1623, inference on supervised fine-tuning model for generation of questions 1533, inference on supervised fine- tuning model for generation of answers 1633, the LLM integration at 1750, or the RL related to diagram 1900 (reinforcement data processor 1970, reinforcement learner 1980, or LLM updates 1990).

[0132] At 2030, the method (or apparatus) may identifying a triggering event associated with a particular context. The triggering event, in some aspects, may include one of a predicted event, a query, or a detected event. In some aspects, the response may include one of a prompt for a user to respond to one of the predicted event or the detected event or a response to the query. Ln some aspects, the response may include an action to respond to and / or mitigate a detected and / or predicted event. For example, referring to FIGs. 1, 8, 9, 14, 18, and 19, the apparatus and / or method may identify and / or generate prompts such as agent prompt 121 / 921, one or more user prompts 122 / 922, question prompt 1820 that may be associated with a particular context.

[0133] At 2040, the method (or apparatus) may generate a response to the triggering event using a context-specific LLM corresponding to the particular context. For example, referring to FIGs. 1, 8, 9, 14, 18, and 19, the apparatus and / or method may generate an output using LLM 170 / 870 / 970 (e.g., operational LLM 171 / 871 / 971 or business LLM 172 / 872 / 972), private LLM answer generator 1432, answer 1880, and / or 1950 based on the triggering event.

[0134] In some aspects, the method (or apparatus) may receive at least one of feedback regarding the response or additional content and context information associated with the particular context. In some aspects, based on receiving one or more of the feedback and / or additional context and / or context, the method may return to 2020 where it may update the context-specific LLM corresponding to the particular context based on at least one of the feedback or the additional content and context information, identifying an additional triggering event associated with the particular context at 2030, and generating, after updating the contextspecific LLM corresponding to the particular context, an additional response to the additional triggering event using the context-specific LLM corresponding to the particular context at 2040. For example, referring to FIGs. 1, 8, 9, 14, 18, and 19, the apparatus and / or method may identify and / or generate an additional prompt such as agent prompt 121 / 921, one or more user prompts 122 / 922, question prompt 1820 that may be associated with a particular context or, referring to FIGs. 1, 8, 9, and 19, the apparatus and / or method may receive feedback associated with the reinforcement learner 180 / 880 / 980 / 1980.

[0135] FIG. 21 is a flow diagram 2100 illustrating a method of generating output associated with an industrial application in accordance with some aspects of the disclosure. In some aspects, the method is performed by a Generative Al appliance or related apparatus and / orcomponents of an industrial application (e.g., implemented by one or more computer devices such as computer device 2205) that performs various analyses and / or processing operations and generates various recommendation and / or responses. At 2110, the apparatus may extract content and context information associated with the industrial application. In some aspects, the content and context information may include information regarding a set of workflows associated with the industrial application. Each workflow in the set of workflows, in some aspects, may include one or more user interfaces (e.g., UIs) of the industrial application, one or more selections via the one or more user interfaces, one or more entries via the one or more user interfaces, or one or more documents associated with the one or more user interfaces of the industrial application. In some aspects, the content information regarding the particular workflow may include, first content provided by the industrial application via the set of one or more user interfaces, data provided to the industrial application via the set of one or more user interfaces, and / or a set of documents provided to the industrial application via the set of one or more user interfaces. For example, referring to FIGs. 1-12 and 14-19, the Gen Al Ul / UX 112 may extract content and / or context information (e.g., at 210, 1710-1730) associated with an industrial application 110.

[0136] In some aspects, the context information regarding a particular workflow in the set of workflows may include a set of one or more user interfaces that are accessed by a user and a set of selections made via the set of one or more user interfaces. The context information regarding the particular workflow, in some aspects, may further include navigation information regarding a navigation through the set of one or more user interfaces via the set of selections. In some aspects, the context information may be used to generate a tree graph (an in-context tree) indicating relationships between a plurality of user interfaces associated with the industrial application. For example, referring to FIGs. 2-6, the content and / or context information may be used to generate the in-context tree 440 / 540.

[0137] The one or more context-specific LLMs, in some aspects, may be associated with a first embedding associated with a current word, a second embedding associated with a current word position, and a third embedding associated with a knowledge graph based on the tree graph, and where the one or more context-specific LLMs use one of (1) a first superposition of the first embedding, the second embedding, and the third embedding, (2) the third embedding and a second superposition of the first embedding and the second embedding, or (3) the second superposition and a third superposition of the second embedding and the third embedding. Forexample, referring to FIGs. 1 , 8-12, and 17-19, the apparatus and / or method may be associated with common embeddings 132 / 932, 143 / 943, 156 / 856, 166 / 866, 168 / 868, the embeddings illustrated in diagrams 1000, 1100, and 1200, or in relation to content embedding generation and / or mapping at 1720.

[0138] At 2120, the method (or apparatus) may identify, produce, or update one or more context-specific LLMs (by adjusting, if producing and / or updating the context-specific LLM) one or more LLMs based on the content and context information. In some aspects, the one or more context-specific LLMs may include at least a first context- specific LLM for a first context and a second context-specific LLM for a second context. The first context, in some aspects, may be associated with a business context and the second context may be an operational context. In some aspects, the one or more context-specific LLMs may include a plurality of user-specific LLMs for a corresponding plurality of users. In some aspects, as part of identifying, producing, or updating one or more context-specific LLMs, the method (or apparatus), at 2125, may produce a plurality of user-specific LLMs associated with a first context-specific LLM by separately adjusting the first context-specific LLM for each of a corresponding plurality of users associated with a first context. In some aspects, producing a particular user-specific LLM of the plurality of user-specific LLMs for a particular user in the corresponding plurality of users may include adjusting the first context-specific LLM based one or more of a role associated with the particular user, a status associated with the particular user, a set of stored actions associated with the particular user, or an application associated with the particular user. For example, referring to FIGs. 1 , 9, and 13-19, the apparatus and / or method may be associated with an initial LLM 1310 or pretrained LLM 1401, pubhc API 1512, public API 1612 with an adjustment associated with one or more of a fine-tuning 157 / 857, 167 / 867, fine-tuning model 1320, any of supervised fine-tuning model 1523 or 1623, inference on supervised fine-tuning model for generation of questions 1533, inference on supervised finetuning model for generation of answers 1633, the LLM integration at 1750, or the RL related to diagram 1900 (reinforcement data processor 1970, reinforcement learner 1980, or LLM updates 1990).

[0139] At 2130, the method (or apparatus) may identifying a triggering event associated with a particular context. The triggering event, in some aspects, may include one of a predicted event, a query, or a detected event. In some aspects, the response may include one of a prompt for a user to respond to one of the predicted event or the detected event or a response to thequery . For example, referring to FIGs. 1 , 8, 9, 14, 18, and 19, the apparatus and / or method may identify and / or generate prompts such as agent prompt 121 / 921, one or more user prompts 122 / 922, question prompt 1820 that may be associated with a particular context.

[0140] At 2140, the method (or apparatus) may generate a response to the triggering event using a context-specific LLM corresponding to the particular context. For example, referring to FIGs. 1, 8, 9, 14, 18, and 19, the apparatus and / or method may generate an output using LLM 170 / 870 / 970 (e.g., operational LLM 171 / 871 / 971 or business LLM 172 / 872 / 972), private LLM answer generator 1432, answer 1880, and / or 1950 based on the triggering event.

[0141] At 2150, the method (or apparatus) may receive at least one of feedback regarding the response or additional content and context information associated with the particular context. In some aspects, based on receiving one or more of the feedback and / or additional context and / or context, the method may return to 2120 where it may update the context-specific LLM corresponding to the particular context based on at least one of the feedback or the additional content and context information, identifying an additional triggering event associated with the particular context at 2130, and generating, after updating the contextspecific LLM corresponding to the particular context, an additional response to the additional triggering event using the context-specific LLM corresponding to the particular context at 2140. For example, referring to FIGs. 1, 8, 9, 14, 18, and 19, the apparatus and / or method may identify and / or generate an additional prompt such as agent prompt 121 / 921, one or more user prompts 122 / 922, question prompt 1820 that may be associated with a particular context or, referring to FIGs. 1, 8, 9, and 19, the apparatus and / or method may receive feedback associated with the reinforcement learner 180 / 880 / 980 / 1980.

[0142] In some aspects, the Generative Al appliance / companion discussed above may provide self-monitoring and business support and automation. For example, the appliance, in some aspects, may automatically monitor its own performance and suggest corrective action if necessary or asked by user. It can also be used to automate tasks, such as generating reports or answering business questions. This can free up human resources both business and IT to focus on other tasks. The Generative Al appliance / companion discussed above, in some aspects, may capable of fast deployment. In some aspects, the appliance is designed to be deployed quickly and easily with hardware, software, Generative Al having both business and operational readiness to fine-tune (or fine-tuned system), without the need for specialized Generative Alskills or training This makes it a good option for businesses that need to get up and tunning quickly.

[0143] In some aspects, the Generative Al appliance / companion discussed above may provide user-friendly interactions. The appliance, in some aspects, may have a user-friendly interface that makes it easy for users to interact with it. This can help to reduce the learning curve and improve adoption. The Generative Al apphance / comp anion discussed above, in some aspects, may provide UI / UX that is user-context centric. For example, in some aspects, the appliance can understand the context of a user’s business request, infrastructure setup, appliance setup and operational conditions to generate suggestions that is relevant to the context. This can help to improve the generated result accuracy and usefulness of the solutions that user need. In some aspects, the Generative Al appliance / companion discussed above may provide easy context switching. For example, the appliance, in some aspects, may easily switch between different contexts, such as generating different types of content or responding to different types of queries via verticalized fine-tuning without changing an underlying LLM, software stack, and hardware. This can help to improve the efficiency of the apphance. In line with the benefits described above, the Generative Al appliance / companion discussed above may provide may easily be adapted to new business. The apphance, in some aspects, may be easily adapted to new business needs or new business processes. This can help businesses to stay ahead of the competition.

[0144] FIG. 22 illustrates an example computing environment with an example computer device suitable for use in some example implementations. Computer device 2205 in computing environment 2200 can include one or more processing units, cores, or processors 2210, memory 2215 (e.g., RAM, ROM, and / or the like), internal storage 2220 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or IO interface 2225, any of which can be coupled on a communication mechanism or bus 2230 for communicating information or embedded in the computer device 2205. IO interface 2225 is also configured to receive images from cameras or provide images to projectors or displays, depending on the desired implementation.

[0145] Computer device 2205 can be communicatively coupled to input / user interface 2235 and output device / interface 2240. Either one or both of the input / user interface 2235 and output device / interface 2240 can be a wired or wireless interface and can be detachable. Input / user interface 2235 may include any device, component, sensor, or interface, physical or virtual, that can be used to provide input (e.g., buttons, touch-screen interface, keyboard, apointing / cursor control, microphone, camera, braille, motion sensor, accelerometer, optical reader, and / or the like). Output device / interface 2240 may include a display, television, monitor, printer, speaker, braille, or the like. In some example implementations, input / user interface 2235 and output device / interface 2240 can be embedded with or physically coupled to the computer device 2205. In other example implementations, other computer devices may function as or provide the functions of input / user interface 2235 and output device / interface 2240 for a computer device 2205.

[0146] Examples of computer device 2205 may include, but are not limited to, highly mobile devices (e.g., smartphones, devices in vehicles and other machines, devices carried by humans and animals, and the like), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, and the like), and devices not designed for mobility (e.g., desktop computers, other computers, information kiosks, televisions with one or more processors embedded therein and / or coupled thereto, radios, and the like).

[0147] Computer device 2205 can be communicatively coupled (e.g., via IO interface 2225) to external storage 2245 and network 2250 for communicating with any number of networked components, devices, and systems, including one or more computer devices of the same or different configuration. Computer device 2205 or any connected computer device can be functioning as, providing services of, or referred to as a server, client, thin server, general machine, special-purpose machine, or another label.

[0148] IO interface 2225 can include but is not limited to, wired and / or wireless interfaces using any communication or IO protocols or standards (e.g., Ethernet, 2202.1 lx, Universal System Bus, WiMax, modem, a cellular network protocol, and the like) for communicating information to and / or from at least all the connected components, devices, and network in computing environment 2200. Network 2250 can be any network or combination of networks (e.g., the Internet, local area network, wide area network, a telephonic network, a cellular network, satellite network, and the like).

[0149] Computer device 2205 can use and / or communicate using computer-usable or computer readable media, including transitory media and non-transitory media. Transitory media include transmission media (e.g., metal cables, fiber optics), signals, carrier waves, and the like. Non-transitory media include magnetic media (e.g., disks and tapes), optical media(e.g., CD ROM, digital video disks, Blu-ray disks), solid-state media (e.g., RAM, ROM, flash memory, solid-state storage), and other non-volatile storage or memory.

[0150] Computer device 2205 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some example computing environments. Computer-executable instructions can be retrieved from transitory media, and stored on and retrieved from non-transitory media. The executable instructions can originate from one or more of any programming, scripting, and machine languages (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, and others).

[0151] Processors) 2210 can execute under any operating system (OS) (not shown), in a native or virtual environment. One or more applications can be deployed that include logic unit 2260, application programming interface (API) unit 2265, input unit 2270, output unit 2275, and inter -unit communication mechanism 2295 for the different units to communicate with each other, with the OS, and with other applications (not shown). The described units and elements can be varied in design, function, configuration, or implementation and are not limited to the descriptions provided. Processors) 2210 can be in the form of hardware processors such as central processing units (CPUs) or in a combination of hardware and software units.

[0152] In some example implementations, when information or an execution instruction is received by API unit 2265, it may be communicated to one or more other units (e.g., logic unit 2260, input unit 2270, output unit 2275). In some instances, logic unit 2260 may be configured to control the information flow among the units and direct the services provided by API unit 2265, the input unit 2270, the output unit 2275, in some example implementations described above. For example, the flow of one or more processes or implementations may be controlled by logic unit 2260 alone or in conjunction with API unit 2265. The input unit 2270 may be configured to obtain input for the calculations described in the example implementations, and the output unit 2275 may be configured to provide an output based on the calculations described in example implementations.

[0153] Processors) 2210 can be configured to extract content and context information associated with the industrial application. The processor (s) 2210 can be configured to produce one or more context-specific large language models (LLMs) by adjusting one or more LLMs based on the content and context information. The processors) 2210 can be configured to identify a triggering event associated with a particular context. The processor(s) 2210 can beconfigured to generate a response to the triggering event using a context-specific LLM corresponding to the particular context. The processors) 2210 can be configured to produce a plurality of user-specific LLMs associated with a first context-specific LLM by separately adjusting the first context-specific LLM for each of a corresponding plurality of users associated with a first context.

[0154] The processors) 2210 can also be configured to receive at least one of feedback regarding the response or additional content and context information associated with the particular context. The processors) 2210 can also be configured to update the context- specific LLM corresponding to the particular context based on at least one of the feedback or the additional content and context information. The processor(s) 2210 can also be configured to identify an additional triggering event associated with the particular context. The processor(s) 2210 can also be configured to generate, after updating the context-specific LLM corresponding to the particular context, an additional response to the additional triggering event using the context- specific LLM corresponding to the particular context.

[0155] Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the essence of their innovations to others skilled in the art. An algorithm is a series of defined steps leading to a desired end state or result. In example implementations, the steps carried out require physical manipulations of tangible quantities for achieving a tangible result.

[0156] Unless specifically stated otherwise, as apparent from the discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing,” “computing,” “calculating,” “determining,” “displaying,” or the like, can include the actions and processes of a computer system or other information processing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system’s registers and memories into other data similarly represented as physical quantities within the computer system’s memories or registers or other information storage, transmission or display devices.

[0157] Example implementations may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may include one or more general-purpose computers selectively activated or reconfigured byone or more computer programs. Such computer programs may be stored in a computer readable medium, such as a computer readable storage medium or a computer readable signal medium. A computer readable storage medium may involve tangible mediums such as, but not limited to optical disks, magnetic disks, read-only memories, random access memories, solid- state devices, and drives, or any other types of tangible or non-transitory media suitable for storing electronic information. A computer readable signal medium may include mediums such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Computer programs can involve pure software implementations that involve instructions that perform the operations of the desired implementation.

[0158] Various general-purpose systems may be used with programs and modules in accordance with the examples herein, or it may prove convenient to construct a more specialized apparatus to perform desired method steps. In addition, the example implementations are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the example implementations as described herein. The instructions of the programming language(s) may be executed by one or more processing devices, e.g., central processing units (CPUs), processors, or controllers.

[0159] As is known in the art, the operations described above can be performed by hardware, software, or some combination of software and hardware. Various aspects of the example implementations may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software), which if executed by a processor, would cause the processor to perform a method to carry out implementations of the present application. Further, some example implementations of the present application may be performed solely in hardware, whereas other example implementations may be performed solely in software. Moreover, the various functions described can be performed in a single unit, or can be spread across a number of components in any number of ways. When performed by software, the methods may be executed by a processor, such as a general-purpose computer, based on instructions stored on a computer readable medium. If desired, the instructions can be stored on the medium in a compressed and / or encrypted format.

[0160] Moreover, other implementations of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the teachings of the present application. Various aspects and / or components of the described example implementations may be used singly or in any combination. It is intended that the specification and example implementations be considered as examples only, with the true scope and spirit of the present application being indicated by the following claims.

Claims

CLAIMSWhat is claimed is:

1. A method of generating output associated with an industrial application, comprising: extracting content and context information associated with the industrial application; producing one or more context-specific large language models (LLMs) by adjusting one or more LLMs based on the content and context information; identifying a triggering event associated with a particular context; and generating a response to the triggering event using a context-specific LLM corresponding to the particular context.

2. The method of claim 1, wherein the one or more context-specific LLMs comprise at least a first context-specific LLM for a first context and a second context-specific LLM for a second context.

3. The method of claim 2 wherein the first context is associated with a business context and the second context is an operational context.

4. The method of claim 1, wherein the one or more context-specific LLMs comprise a plurality of user-specific LLMs for a corresponding plurality of users.

5. The method of claim 1 further comprising: producing a plurality of user-specific LLMs associated with a first context-specific LLM by separately adjusting the first context-specific LLM for each of a corresponding plurality of users associated with a first context.

6. The method of claim 5 , wherein producing a particular user-specific LLM of the plurality of user-specific LLMs for a particular user in the corresponding plurality of users comprises adjusting the first context-specific LLM based one or more of: a role associated with the particular user; a status associated with the particular user; a set of stored actions associated with the particular user; or an application associated with the particular user.

7. The method of claim 1 , wherein the content and context information comprises information regarding a set of workflows associated with the industrial application, wherein each workflow in the set of workflows includes one or more user interfaces of the industrial application, one or more selections via the one or more user interfaces, one or more entries via the one or more user interfaces, or one or more documents associated with the one or more user interfaces of the industrial application.

8. The method of claim 7, wherein the context information regarding a particular workflow in the set of workflows comprises a set of one or more user interfaces that are accessed by a user and a set of selections made via the set of one or more user interfaces.

9. The method of claim 8, wherein the context information regarding the particular workflow further comprises navigation information regarding a navigation through the set of one or more user interfaces via the set of selections.

10. The method of claim 9, wherein the context information is used to generate a tree graph indicating relationships between a plurality of user interfaces associated with the industrial application.

11. The method of claim 10, wherein the one or more context-specific LLMs are associated with a first embedding associated with a current word, a second embedding associated with a current word position, and a third embedding associated with a knowledge graph based on the tree graph, and wherein the one or more context-specific LLMs use one of (1) a first superposition of the first embedding, the second embedding, and the third embedding, (2) the third embedding and a second superposition of the first embedding and the second embedding, or (3) the second superposition and a third superposition of the second embedding and the third embedding.

12. The method of claim 8, wherein the content information regarding the particular workflow comprises, first content provided by the industrial application via the set of one or more user interfaces, data provided to the industrial application via the set of one or more user interfaces, and a set of documents provided to the industrial application via the set of one or more user interfaces.

13. The method of claim 1, wherein the triggering event comprises one of a predicted event, a query, or a detected event.

14. The method of claim 13, wherein the response comprises one of a prompt for a user to respond to one of the predicted event or the detected event or a response to the query.

15. The method of claim 1 further comprising: receiving at least one of feedback regarding the response or additional content and context information associated with the particular context; updating the context-specific LLM corresponding to the particular context based on at least one of the feedback or the additional content and context information; identifying an additional triggering event associated with the particular context; and generating, after updating the context-specific LLM corresponding to the particular context, an additional response to the additional triggering event using the context-specific LLM corresponding to the particular context.

16. An apparatus for generating output associated with an industrial application, comprising: at least one memory; a user interface; and at least one processor coupled to the at least one memory and, based at least in part on stored information that is stored in the at least one memory, the at least one processor, individually or in any combination, is configured to: extract content and context information associated with the industrial application; identify one or more context-specific large language models (LLMs) based on the content and context information; identify a triggering event associated with a particular context; and generate a response to the triggering event using a context-specific LLM corresponding to the particular context.

17. The apparatus of claim 16, wherein the one or more context-specific LLMs comprise at least a first context-specific LLM for a first context and a second context-specific LLM for a second context, wherein the first context is associated with a business context and the second context is an operational context.

18. The apparatus of claim 16, wherein the one or more context-specific LLMs comprise a plurality of user-specific LLMs for a corresponding plurality of users.

19. The apparatus of claim 16, wherein the content and context information comprises information regarding a set of workflows associated with the industrial application, wherein each workflow in the set of workflows includes one or more user interfaces of the industrial application, one or more selections via the one or more user interfaces, one or more entries via the one or more user interfaces, or one or more documents associated with the one or more user interfaces of the industrial application, wherein the context information regarding a particular workflow in the set of workflows comprises a set of one or more user interfaces that are accessed by a user and a set of selections made via the set of one or more user interfaces, wherein the context information regarding the particular workflow further comprises navigation information regarding a navigation through the set of one or more user interfaces via the set of selections, wherein the context information is used to generate a tree graph indicating relationships between a plurality of user interfaces associated with the industrial application, and wherein the one or more context-specific LLMs are associated with a first embedding associated with a current word, a second embedding associated with a current word position, and a third embedding associated with a knowledge graph based on the tree graph, and wherein the one or more context-specific LLMs use one of (1) a first superposition of the first embedding, the second embedding, and the third embedding, (2) the third embedding and a second superposition of the first embedding and the second embedding, or (3) the second superposition and a third superposition of the second embedding and the third embedding.

20. A method of generating output associated with an industrial application, comprising: extracting content and context information associated with the industrial application; identifying one or more context-specific large language models (LLMs) based on the content and context information; identifying a triggering event associated with a particular context; and generating a response to the triggering event using a context-specific LLM corresponding to the particular context.

Images