Large language model (LLM) message generation

Abstract

A large language model (LLM) message generation system and method tor generating a response to a user message. The method includes: obtaining a text document specified by a user; chunking the text document into a plurality of chunks; generating text gloss data for each of the plurality of chunks based on the chunk and a predetermined prompt; storing the text gloss data for each chunk into a vector data store along with an identifier for the chunk and / or tine chunk itself; receiving a user message; querying the vector data store with a vector data store query, wherein the vector data store query is generated based on the user message; obtaining chunk(s) based on querying the vector data store with the vector data store query; generating a language model input based on the one or more identified chunks; and generating a response message based on inputting the language model input.

Description

LARGE LANGUAGE MODEL (LLM) MESSAGE GENERATION TECHNICAL FIELD

[0001] This disclosure relates to text message generation and large language models (LLMs) as well as generativepre-trained transformer models. BACKGROUND

[0002] Recently, natural language processing (NLP) has soared in popularity thanks to its impressive natural languageunderstanding (NLU) and natural language generation (NLG) capabilities, particularly due to the transformer technology set forth in Vaswani, Ashish, et al. “Attention is all you need.” Advances in neural information processing systems 30 (2017). Pretraining a large language model (LLM) involves a large amount of resources, and generally costs in the millions of dollars purely in operating costs, not to mention that such pretraining generally involves hundreds or thousands of the latest specialized graphics processing units (GPUs), such as the Nvidia H100 Tensor Core™ GPU, and still takes hundreds or thousands of hours. On the other hand, once a model is pretrained, fine-tuning may be performed, such as where a pretrained LLM is further refined to solve a particular task, such as sentiment analysis, question answering, summarization, generation, etc. This generally takes 100 hours and on the order of ten (10) H100 Tensor Core™ GPUs, costing thousands of dollars. In-context learning (ICL) is used to learn a small amount of information; more generally, ICL refers the ability of a language model to adapt to new information presented in the current conversation or context, even if it is not permanently incorporated into the model’s knowledge base.

[0003] Fine-tuned pretrained LLMs based on such transformer technology, including generative pretrainedtransformers (GPT), may be prompt-based models such, as InstructGPT, that take, as input, a prompt that is input into the LLM and that may be formed as a question, statement, command, or combination thereof; generally, in many instances, certain predefined prompt field identifiers are used consistently as part of training, sometimes where the predefined prompt field identifiers are specially-selected for a particular task. Prompts are a powerful tool because LLMs may be trained generally on a large corpus of text and information, but then specially adapted to incorporate customized context, via prompting, into the input at runtime thereby providing a specially-tailored output built through leveraging the power of large, pretrained LLMs in a manner tuned to the customized context.

[0004] Retrieval augmentation techniques have been used to retrieve information useful or relevant for generating asuitable or desired response to a user message through use of a LLM. However, retrieval augmentation techniques are problematic, especially when data being indexed for retrieval is large, unstructured, and varying in kind. SUMMARY

[0005] According to a first aspect, there is provided a method of generating a response to a user message. Themethod includes: obtaining a text document specified by a user, wherein the text document is a document thathas text; chunking the text document into a plurality of chunks; generating text gloss data for each chunk of the plurality of chunks based on inputting the chunk along with a predetermined text gloss generation prompt into a large language model; storing the text gloss data for each chunk of the plurality of chunks into a vector data store along with an identifier for the chunk and / or the chunk itself; receiving a user message; querying the vector data store with a vector data store query, wherein the vector data store query is generated based on the user message; obtaining one or more chunks based on an output resulting from querying the vector data store with the vector data store query; generating a language model input based on the one or more obtained chunks; and generating a response message based on inputting the language model input into the large language model or another large language model.

[0006] According to embodiments, the method includes any one or more, including any technically-feasiblecombination, of the following: for each chunk of the plurality of chunks, a text gloss language model input is generated, and wherein the text gloss language model input includes the chunk and the predetermined text gloss generation prompt; for each chunk of the plurality of chunks, the text gloss language model input further includes a portion of the text document that is different from the chunk and / or content based on the text document that is different from the chunk; for each chunk of the plurality of chunks, the text gloss language model input includes a summary of the text document or text content from the text document that includes content of at least one other chunk than the chunk; further comprising: obtaining an image from the text document; generating text gloss data for the image based on inputting the image into the large language model or another vision-enabled large language model; and storing the text gloss data for the image into the vector data store; the predetermined text gloss generation prompt includes text instructing the large language model to generate a structured output that contains a list of glossary terms or phrases relevant to subject content identified as the chunk; and / or at least one vector data store entry for each chunk of the plurality of chunks is generated based on parsing the structured output to obtain the text gloss data for the chunk.

[0007] According to yet another aspect, there is provided a method of generating a response to a user message. Themethod includes: obtaining a text document specified by a user, wherein the text document is a document that has text; storing the text document in a content data store; chunking the text document into a plurality of chunks; generating text gloss data for each chunk of the plurality of chunks based on inputting the chunk along with a predetermined text gloss generation prompt into a large language model; storing the text gloss data for each chunk of the plurality of chunks into a vector data store along with an identifier for the chunk; receiving a user message; querying the vector data store with a vector data store query, wherein the vector data store query is generated based on the user message; obtaining one or more chunks based on one or more chunk identifiersreturned from querying the vector data store with the vector data store query; generating a language model input based on the one or more obtained chunks; and generating a response message based on inputting the language model input into the large language model or another large language model.

[0008] According to embodiments, the method includes any one or more, including any technically-feasiblecombination, of the following: for each chunk of the plurality of chunks, a text gloss language model input is generated, and wherein the text gloss language model input includes the chunk and the predetermined text gloss generation prompt; for each chunk of the plurality of chunks, the text gloss language model input further includes a portion of the text document that is different from the chunk and / or content based on the text document that is different from the chunk; for each chunk of the plurality of chunks, the text gloss language model input includes a summary of the text document or text content from the text document that includes content of at least one other chunk than the chunk further comprising: obtaining an image from the text document; generating text gloss data for the image based on inputting the image into the large language model or another vision-enabled large language model; and storing the text gloss data for the image into the vector data store; the predetermined text gloss generation prompt includes text instructing the large language model to generate a structured output that contains a list of glossary terms or phrases relevant to subject content identified as the chunk; and / or at least one vector data store entry for each chunk of the plurality of chunks is generated based on parsing the structured output to obtain the text gloss data for the chunk.

[0009] According to yet another aspect, there is provided a method of generating a response to a user message. Themethod includes: obtaining a text document specified by a user, wherein the text document is a document that has text; storing the text document in a content data store; chunking the text document into a plurality of chunks; generating text gloss data for each chunk of the plurality of chunks based on inputting the chunk along with a predetermined text gloss generation prompt into a large language model; storing the text gloss data for each chunk of the plurality of chunks into a vector data store along with an identifier for the chunk; receiving a user message; generating a vector data store query based on an output generated by inputting the user message into the large language model or a second language model; querying the vector data store with the vector data store query; obtaining one or more chunks based on one or more chunk identifiers returned from querying the vector data store with the vector data store query; generating a language model input based on the one or more obtained chunks; and generating a response message based on inputting the language model input into the large language model or another large language model.

[0010] According to embodiments, the method includes any one or more, including any technically-feasiblecombination, of the following:the vector data store query is determined by parsing the user message to identify one or more text strings to be used for querying the vector data store; and / or storing the text document into a content data store, wherein the one or more chunks are obtained by querying the content data store with the one or more chunk identifiers returned from querying the vector data store. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Preferred exemplary embodiments will hereinafter be described in conjunction with the appended drawings,wherein like designations denote like elements, and wherein:

[0012] FIG. 1 is a schematic block diagram depicting an embodiment of a large language model (LLM) messagegeneration system, including a content glossary network and vector data store, according to at least one embodiment;

[0013] FIG.2 is a schematic block diagram depicting an embodiment of a LLM message generation system, includinga knowledge extractor network, user knowledge data store, top-subject application programming interface (API) subsystem, and topic extractor network, according to at least one embodiment;

[0014] FIG.3A is a schematic block diagram depicting an embodiment of the LLM message generation system of FIG.2, but further including a distributed training framework for fine-tuning or otherwise training a plurality of head networks as well as a feedback generation data network, according to at least one embodiment;

[0015] FIG.3B is a schematic block diagram depicting an embodiment of the LLM message generation system of FIG.2, but further including a distributed training framework for fine-tuning or otherwise training a plurality of low- rank models as well as a feedback generation data network, according to at least one embodiment;

[0016] FIG.4 is a schematic block diagram depicting an embodiment of the LLM message generation system of FIG.2, but further including an autoregressive target generation network and a target-topic data store, according to at least one embodiment;

[0017] FIG.5 is an exemplary schematic depicting an example of a tree with outputs (intermediate targets) generatedusing an autoregressive target generation process having a recursion level or depth set to three (3), according to one embodiment;

[0018] FIG.6 is a schematic block diagram depicting an embodiment of a LLM message generation system having amessage generation network comprised of a base model and low-rank model, as well as a custom low-rank model training framework, according to at least one embodiment;

[0019] FIG.7 is a schematic block diagram depicting an embodiment of the LLM message generation system of FIG.6, and further including an application-specific trainer that may be used for training one or more of the message generation networks discussed herein, according to at least one embodiment;

[0020] FIG.8 is a flowchart illustrating an exemplary method of training a low-rank model using custom training data,according to at least one embodiment;

[0021] FIG.9 is a flowchart illustrating an exemplary method of training a plurality of low-rank models using customtraining data and a shared training phase, according to at least one embodiment;

[0022] FIG.10 is a flowchart illustrating another exemplary method of training a low-rank model using custom trainingdata, according to at least one embodiment;

[0023] FIG. 11 is a flowchart illustrating yet another exemplary method of training a low-rank model using customtraining data, according to at least one embodiment;

[0024] FIG. 12 is a schematic block diagram depicting an embodiment of a LLM message generation system thatimplements a dual-selectable ensemble network according to an outer head configuration, according to at least one embodiment;

[0025] FIG. 13 is a schematic block diagram depicting an embodiment of a LLM message generation system thatimplements a dual-selectable ensemble network according to an inner head configuration, according to at least one embodiment;

[0026] FIG. 14 is a flowchart illustrating an exemplary method of generating a response message based on aninference output, according to at least one embodiment;

[0027] FIG.15 is a schematic diagram depicting an exemplary graphical user interface (GUI) that may be used as apart of a client or user device, according to at least one embodiment;

[0028] FIG.16 is a flowchart illustrating an exemplary method of generating a response message based on queryinga vector database having text gloss data, according to at least one embodiment;

[0029] FIG.17 is a schematic diagram illustrating an embodiment of a message generation system implemented usingPython for its backend and JavaScript for its front-end, according to at least one embodiment;

[0030] FIG. 18 is a flowchart illustrating an exemplary method of generating a response message in response to auser message, according to at least one embodiment;

[0031] FIG.19 is a flowchart illustrating an exemplary method of ingesting content and generating content gloss datafor the ingested content, according to at least one embodiment;

[0032] FIG. 20 is a flowchart illustrating an exemplary method of generating a response message in response to auser message, according to at least one embodiment; and

[0033] FIG.21 is a flowchart illustrating an exemplary method of obtaining content to be included in a language modelinput, according to at least one embodiment. DETAILED DESCRIPTION

[0034] According to one aspect, there is provided a system and method for generating a response message for auser message, such as in the context of a chatbot. In embodiments, the method includes: receiving a user message from a user; querying a vector database for a content identifier based on the user message, wherein the vector database includes textual gloss data for content; obtaining a portion of the content based on the content identifier; generating a language model input based on the user message and the portion of the content;and generating a response message based on inputting the language model input into a language model whereby the language model performs inference using the language model input.

[0035] According to embodiments, the system includes a retrieval augmentation feature whereby information usefulfor generating a response message is retrieved. The retrieval augmentation feature includes querying a vector database for information relevant to a user message, such as the most-recently received user message when used in the context of a chatbot, for example. The retrieval augmentation feature includes querying a glossary vector database for a content identifier that is used to uniquely identify terms or textual glosses (referred to as “textual gloss data”) to which the user message relates, and the result of this query is referred to as a glossary vector query. The results of the glossary vector query include one or more content identifiers, each of which is associated with textual gloss data. The one or more content identifiers are then used to obtain content whereby this obtained content is associated with the textual gloss data as the textual gloss data is representative of subject matter described and / or within the obtained content.

[0036] Generation of Textual gloss data. The textual gloss data includes one or more terms or words that are pertinentto a body of content, such as an unstructured text document or other unstructured text data. The unstructured text document may be a book, a patent, a technical document, a magazine, a newspaper, a blog post, an email, a transcript, a research paper, a social media post, a forum comment, a legal contract, a script, a website, a newsletter, a user review, or any other form of written content without a predefined data model or organized format. The unstructured text data may be processed to provide one or more content identifiers, each of which is associated with content of the unstructured text data. This content may be partitioned or split according to a number of characters or tokens (e.g., sub-word tokens, words) so that for a given document (or given unstructured text input) one or more unstructured text entries are obtained. Here, it is possible that the unstructured text input is small enough in length that it is below the number of characters or tokens used for partitioning the data, and so only one unstructured text entry is provided. However, according to at least some implementations, the unstructured text input is large in terms of character / token length, resulting in a multitude of unstructured text entries (i.e., 100 or more unstructured text entries); indeed, according to some implementations, the number of unstructured text entries is 1000 or more, ten thousand or more, one hundred thousand or more, or one million or more. That is, according to embodiments, the disclosed glossary term generation technique is suitable for processing very large unstructured text inputs, such as very large text documents.

[0037] According to one embodiment, the textual gloss data is automatically generated through inputting content intoa LLM, such as gpt-3.5-turbo or gpt-4 provided by OpenAI™, for example; of course, other GPT-based LLMs, BERT LLMs, or other LLMs may be used, according to embodiments. In one embodiment, a textual gloss data generation prompt is used as input into a GPT-based LLM where the textual gloss data generation prompt includes an unstructured text entry and predefined prompt data. The predefined prompt data is comprised of characters / tokens, such as one or more text sentences, that are configured to instruct the LLM to produce an output having parsable text that includes a list of glossary terms or phrases, which is referred to as the glossary text data. Moreover, the text output provided by the LLM here is generated in a manner so as to be structuredaccording to a predefined output structure, which may be imparted on the output of the LLM through prompt engineering techniques and / or fine-tuning, such as fine-tuning of the LLM itself and / or training of a low-rank adapter or head network.

[0038] In embodiments, a large input LLM, which is an LLM capable of processing a context window or input of atleast 80k tokens, may be used, such as ‘gpt-4-1106-preview’, for example. Use of a large input LLM is desirable for certain use cases, such as technical documents (e.g., patents, patent application pre-grant publications), as it allows for a large portion of the document, oftentimes the entire document, to be passed as input along with the prompt. In the context of generating textual gloss data, this provides an advantage as the construction of the text characterizations or other glosses generated for a given term or concept may take into consideration a large portion of the source document and in a single forward pass during inference of the LLM.

[0039] As mentioned above, the predefined output data is structured, and this predefined output structure is predefinedin terms of its form, notation, or data representation. For example, in one embodiment, Javascript Object Notation (JSON) is used; in other embodiments, different data representations may be used, such as eXtensible Markup Language (XML), JavaScript Object Notation (JSON), Yet Another Markup Language (or sometimes referred to as YAML ain’t markup language) (YAML), Tom's Obvious Minimal Language (TOML), Comma- Separated Values (CSV), INI, Protocol Buffers (protobufs), MessagePack, Concise Binary Object Representation (CBOR), Avro, Parquet, and / or Optimized Row Columnar (ORC). Of course, other serialization or markup languages may be used. The text output of the LLM is then parsed or otherwise processed according to the predefined output structure, such as through using a JSON parser to convert the text output to a JSON object when the predefined output structure is JSON. For example, in python, the “json” module may be used to perform this JSON parsing, such as through json.loads(text_output). And, for example, when XML is used as the predefined output structure, the xml module in python may be used, such as through use of xml_object = xml.etree.ElementTree.fromstring(text_output). This parsing results in a data object useful for separating out portions of the results contained within the text output. More particularly, the glossary terms are extracted along with each’s associated content identifier.

[0040] According to embodiments, a large language model (LLM) message generation system and method areprovided herein, particularly where the LLM message generation system and method are used for generating a response message in reply to a user input, which may include a user message and user-configurable options. More particularly, according to embodiments, the LLM message generation system and method includes a message generation network that is implemented using an LLM, such as an LLM generated based on GPT technology. And, even more particularly, at least according to embodiments, the LLM message generation system and method is a live or real-time message generation system, where response messages are continuously generated in response to receiving user messages and / or other user inputs; for example, a text- generation or chatbot message generation system is used whereby chat messages or response messages are generated by the message generation system for purposes of carrying on a conversation with the user.

[0041] The disclosed systems and methods may be adapted to include any technically-feasible combination of thevarious features disclosed herein, including the swappable head network technology, the swappable low-rankmodel technology, the custom user data low-rank model training technology, the shared low-rank model training technology, the distributed head training technology, the knowledge extraction technology, the knowledge injection technology, the topic extraction technology, the intermediate target generation technology, the target guiding technology, the speech-adapted LLM message generation technology, the audio feature-based transformer technology, the dual-selectable ensemble network technology including distributed training of a dual-selectable ensemble network, and other such features and techniques disclosed below.

[0042] Swappable Network. According to embodiments, the message generation network includes a backbonenetwork (implemented as an LLM) and a plurality of swappable networks (or “swappable models” as each network here corresponds to a model such as a feed forward neural network), each of which is selectable (based on user options, for example) to be used to process output of the backbone network (implemented by a backbone model) so as to generate an output, which may be a tone-conditioned or voice-conditioned message, at least in embodiments. The swappable models may be a plurality of swappable head models or a plurality of low-rank adapter models (referred to as low-rank adapters).

[0043] Swappable Head Network. Each of the swappable head networks is configured so as to be operativelyconnected to an output (or last layer) of the backbone network. In embodiments, at least one of the swappable head networks is a predetermined tone conditioning head network that is comprised of a feed forward neural network that is fine-tuned or otherwise trained according to predetermined tone conditioning data, which indicates a tone to use for conveying a response in the response message—thus, here, the response message is a tone-conditioned response message since the response from the LLM is conditioned using the predetermined tone conditioning data. Thus, according to embodiments, the message generation network is a fine-tuned tone-conditioned message generation network. In other embodiments, the swappable head network is a customized head, which is a head network that is trained based on customized training head data, which is data that is specified by a user for purposes of training a head network and, in embodiments, this data is obtained from the user, such as through uploading the data to a data store hosted by the system. Accordingly, at least in embodiments, the customized tone-conditioned message generation network is a user- customized tone-conditioned message generation network in that the end user is the user having provided the customized head training data.

[0044] According to embodiments, the swappable head network is or at least includes a feed forward neural networkthat takes, as input, a backbone network output and then outputs logits used to provide a probability distribution (obtained using softmax function on the logits, for example) from which tokens are sampled. In embodiments, nucleus sampling and / or bean-search sampling techniques are used.

[0045] Swappable Low-Rank Model. Each of the swappable low-rank models is a neural network that has a smalleror lower rank than a base or body model (e.g., the pretrained LLM), such as an application-specific model (e.g., application-specific instruct model) or other pretrained LLM-based model. When the base model is an LLM, the low-rank model is referred to as a low-rank LLM. In embodiments, the low-rank models are low-rank adapters. In the context of neural networks, the rank of a matrix denotes the maximum count of linearly independent rows or columns within it. This rank serves as an indicator of the information's uniqueness in thematrix. Low-rank approximation techniques, such as the low-rank adaptation discussed in E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S.Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models (2021; arXiv preprint arXiv: 2106.09685) [hereinafter “LoRA for LLM technology”], may be used for compressing a matrix or tensor. A low-rank matrix or tensor offers a representation of the same information with fewer parameters, contributing to memory conservation, computational speed enhancement, and overfitting reduction by limiting model capacity.

[0046] Low-rank adaptation refers to a task-specific parameterization technique that involves adding a low-rank matrixor tensor to the pre-trained model so that an input used by both the base model and the low-rank model (called a low-rank adapter here as it is used in parallel with a base model) to generate a base model output and a low- rank model output that are then combined to form the final output, such as through use of the LoRA for LLM technology. This low-rank model introduces a smaller number of parameters than the base model (has a lower rank), enabling efficient fine-tuning for specific tasks without having to train the pre-trained base model. By employing a low-rank representation, the low-rank model reduces computational complexity and facilitates the adaptation of the pre-trained model to new tasks during inference. The base model is considered to have full- rank, and “rank” here refers to the overall rank of the tensors used by the model. In embodiments, the rank of the low-rank model is set based on the particulars of the use case for which it is to be used. As mentioned, the process of low-rank approximation involves generating a matrix or tensor of lower rank and, consequently, fewer parameters. Oftentimes, this procedure is directed toward maintaining the most pertinent components of the original matrix, usually identified via methods such as singular value decomposition, while eliminating the least important ones. A low rank indicates a simpler, less complex model with a potential for increased generalizability but a possible limitation in fitting more complex patterns in the data. Conversely, a high rank may lead to a model capable of fitting more intricate patterns, but with a higher risk of overfitting and higher computational expenses. Therefore, selecting the appropriate rank or approximating it appropriately forms a critical part of efficient and effective neural network design. Accordingly, the rank of the low-rank model is set based on the particulars of the use case for which it is to be used.

[0047] In embodiments, at least one of the swappable low-rank models is a low-rank model, such as a low-rankadapter, that is fine-tuned or otherwise trained according to predetermined tone conditioning data. Thus, according to embodiments, the message generation network is a fine-tuned tone-conditioned message generation network. In other embodiments, one or more of the swappable low-rank models is a customized low-rank model, which is a low-rank model that is trained based on customized training data, which is data that is based on custom user data specified by a user (e.g., through an application programming interface) for purposes of training a network. In embodiments, this custom user data is obtained from the user, such as through uploading the data to a data store hosted by the system. Accordingly, at least in embodiments, the customized tone-conditioned message generation network is a user-customized tone-conditioned message generation network in that the end user is the user having provided the custom user data used in the training.

[0048] Dual-selectable Ensemble Network. A dual-selectable ensemble network refers to a neural network havingtwo models that are selected models to choose from and that of them are to be used in concert with one another(ensemble) in order to generate a network output. Oftentimes, a base model is used as well in the ensemble network, although the base model is static (not selected at runtime), at least in embodiments. According to embodiments, the system and method herein may employ a dual-selectable ensemble network in which a low- rank model is selected from a plurality of low-rank models and a head network is selected from a plurality of head models. In at least some embodiments, the selected low-rank model and the base model are used to generate an output that is then used as input by the selected head model in order to generate a response message. The base model may be any of a variety of LLMs, such as a post-training quantized LLM (e.g., generative post-training quantization (GPTQ) LLM).

[0049] Custom user data low-rank model training technology. A plurality of low-rank models is capable of beinggenerated, each being tailored to the training data set used for its respective training. While low-rank adaptation has drastically improved the fine-tuning process, it still uses a large number of parameters and generally expends a fair number of resources during the fine-tuning process, which is not suitable for certain implementations. Generally, for such reasons, training has been reserved for being carried out using specialized training data generally involving manual labelling or domain-specific distillation techniques in which labelled training data is autonomously generated, and often only when provision of such resources is balanced with the particular use case—that is, because expending a large number of resources is costly, the underlying use case must make sense from a cost-benefit perspective. Thus, according to embodiments, a low-rank model is generated using a training process that implements double quantization, k-bit (e.g., 4-bit) NormalFloat (NF4) quantization, and / or paged optimizers, referred to as QLoRA when such techniques are employed. It has been demonstrated that such techniques enable super-efficient fine-tuning of low-rank LLMs. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer, QLoRA: Efficient Finetuning of Quantized LLMs, 2023.

[0050] Distributed Training of Swappable Heads. Distributed training of the plurality of swappable heads is performed,according to embodiments. The distributed training includes, assuming H number of swappable head networks are being trained: generating P prompts, each of which corresponds to one of the H swappable head networks; then using the prompts as input into the backbone network; and then passing the output of the backbone network into each of the H swappable head networks so that each swappable head network generates an output based on each of the P prompts—thus, a total of H×P outputs are generated, where there is a one-to- one relationship between prompts P and tones T in the present embodiment. According to embodiments, an actor-critic network is then used to generate an output loss for each of the head networks, and the output losses are then combined using a weighted loss function and backpropagation is used to train the swappable head networks and / or the backbone network—that is, according to embodiments, loss is propagated back through the backbone network and used to adjust parameters of the backbone network and, likewise, loss may be back propagated through each of (or one or more of) the swappable head network so as to fine-tune or adjust the parameters of the swappable head network. In embodiments, the swappable head networks are based on an actor-critic network that uses a specialized critic that is geared toward human language tone conditioning.

[0051] The term “tone” herein is used in the context of communications and, in embodiments, particularly to writtencommunications; in that regard, the term “tone” as used herein refers to both the concepts of “tone” and “voice”, as the concepts “tone” and “voice” are intertwined (nonetheless, different) concepts that shape how a reader (or interpreter) perceives a piece of writing. Generally, voice is the unique style or personality of the writing, reflecting the writer’s unique or distinct way of talking through text, and voice is typically consistent across different works by the same author. Voice may be shaped by the writer’s contextual inputs, and in a more apt sense, by one’s unique perspective, values, and stylistic preferences, including their choice of words, sentence structure, and rhythm. For example, a writer might have a conversational, formal, playful, or authoritative voice. On the other hand, tone refers to the attitude or emotional aspects of the writing. Unlike voice, tone not uncommonly changes significantly within the works of a single author, often influenced by the subject matter or intended audience. A piece of writing may have a serious, humorous, sarcastic, sad, enthusiastic, or critical tone, to name a few examples. For example, a solemn and / or respectful tone might be appropriate for an obituary, while a playful and / or joyous tone might suit a post celebrating a holiday.

[0052] During operation, a swappable head of the plurality of swappable heads is selected based on user-configurableoptions, such as a user selection of “tone” or “voice” attributes, which may be made at the user device through a human-machine interface, such as through receiving user touch input on a touchscreen display corresponding to a displayed graphical user interface (GUI).

[0053] And, in embodiments, a specialized critic is geared toward the user knowledge data for the user, which,according to implementations, results in enhanced personalization so that the response messages are tailored to the user.

[0054] Distributed training of a dual-selectable ensemble network is performed in which the backbone networkdiscussed above is comprised of a base model and a low-rank model. However, during training, low-rank adaptation is used so that weights / parameters of the low-rank model are updated while the weights / parameters of the base model are frozen (not updated). This enables a plurality of head networks to be trained in a custom manner for an already-customized low-rank model-based network, such as the low-rank models discussed herein that use custom user data for training.

[0055] Thus, in embodiments, distributed training of a message generation network having a plurality of heads and asingle low-rank model is performed so that the plurality of heads and the low-rank model are trained simultaneously while the base model weights are kept frozen (unchanged). Use of this distributed training framework enables multiple head networks and the low-rank model to be simultaneously tuned specifically to customer user data, such as conversation history from the user, so that a plurality of custom swappable heads may be generated, such as one for each tone, for example. Moreover, the distributed training low-rank adaptation approach enables use of efficient low-rank adaptation, such as QLoRA approaches, while simultaneously providing for training the base model with each of a plurality of different head networks, all geared for a particular task, such as for transforming the output into one that incorporates a particular, predefined tone. In embodiments, the low-rank model and the base model constitutes the backbone network or model of the message generation network such that the combined output of the low-rank model and thebase model is fed as input into the head networks. Accordingly, in embodiments, a single backbone model is used for distributed training of the plurality of heads.

[0056] As mentioned, the distributed training low-rank adaptation approach enables use of efficient low-rankadaptation while simultaneously providing for training the base model with each of a plurality of different head networks, and this is effected through performing the distributed training for the plurality of heads wherein, during backpropagation, only model weights / parameters of the low-rank model and the head model are updated, while the base model weights / parameters remain frozen or unchanged.

[0057] Knowledge Extractor and Injector. A knowledge extractor is provided, at least in embodiments, wherein theknowledge extractor takes, as input, a user message and extracts knowledge from the user message. For example, the knowledge extractor is an LLM-based pipeline that extracts information from the user message, such as a user’s favorite color, a user’s food preferences, user sentiment toward objects or ideas, etc., and stores the extracted data into the user knowledge data store. In embodiments, the user knowledge data store includes data representing a knowledge graph that is constructed based on the extracted knowledge from the user message.

[0058] Extracted knowledge, or other gathered user knowledge or contextual information, may be used to supplementthe user message when the input or prompt is being built. In embodiments, a topic extractor network is used to determine a topic based on the user message (and / or previous user messages); the topic is used to obtain information about the extracted topic; and then the information is used as input, along with the user message, into the LLM, which then generates the response message. For example, the context of the conversation is determined to relate to desserts (through use of the topic extractor network) and then the knowledge graph for the user is queried to determine what desserts a user has expressed a sentiment toward, and this information is then used to augment the user message so that the LLM may respond to the user message based on context relevant to the user.

[0059] Autoregressive Conversation Target Text Generation. Autoregressive target response text generation is usedto direct a user message to an intermediate target that is determined based on a conversation target, with an intention of directing or nudging the conversation toward the conversation target. The conversation target may be a predefined subject or topic, or may even be a particular object or idea, such as a particular epistemological objective, a product or service, a particular proposition or assertion, etc. The autoregressive target response text generation includes using a LLM (referred to as a target generation LLM), particularly arranged as a network that takes two contextual inputs and outputs a contextual center of those two contextual inputs. In embodiments, the contextual inputs each correspond to a concept or idea (which may be further supplemented by custom context, in embodiments), and the contextual center of those concepts or ideas.

[0060] According to embodiments, the autoregressive target generation network is implemented using two parts: anintermediate target generator and a “recursor” (a function used to control the recursion). The intermediate target generator is used to generate the intermediate target, which may be a topic commonality between two input topics, for example. The recursor is used to control recursion of the intermediate target generator, suchas specifying the depth of recursion and implementing the calls, in a recursive fashion, to the intermediate target generator.

[0061] According to one embodiment, the intermediate target generator is implemented through generating wordembeddings, such as through using Word2Vec, where the input words are directly translated into numerical vectors that represent their semantic meaning in a high-dimensional space. And, in another embodiment, the intermediate target generator is implemented through an LLM.

[0062] In one embodiment, topics (which may be anything from words, phrases, or entire documents) are representedas vectors or tensors encoded into high-dimensional space. A popular model for creating these vector representations is Word2Vec, which is a group of models that generate word embeddings using shallow (two- layer) neural networks trained to reconstruct linguistic contexts of words. Given a large corpus of text as input, Word2Vec outputs a vector space, typically of several hundred dimensions, where each unique word in the corpus receives a corresponding vector. In this space, words sharing common contexts reside near each other. These embeddings capture various linguistic properties like synonyms, analogies, and more. When vectors for distinct terms like “weekend” and “barbecue sauce” are present, their commonality can be discovered in this high-dimensional space. This exploration is usually done by examining vectors close to them in the space; nearby vectors representing other words or concepts can suggest abstract or indirect commonalities. For instance, if the vectors for “weekend” and “barbecue sauce” are both proximal to vectors for words such as “leisure”, “party”, “outdoor”, these could be inferred as common themes. For more complex relationships, models like BERT or GPT could be used for encoding the vectors, as such models are equipped to understand intricate semantic relationships and might be employed to extract more abstract commonalities. It is important to note that the effectiveness of these methodologies can significantly depend on the specificity of the data and the quality of the vector representations.

[0063] The autoregressive target generation network includes two main components: the commonality function(producing a regressive result, which may be a word or a vector) and the recursor. In general, the commonality function determines the commonality (also referred to as the “contextual center”) and, in embodiments, is implemented as a function that returns a word or words representing the commonality after determining the commonality. In this implementation, this commonality function is implemented as a part of a function that takes two terms (two inputs, such as two words, terms, or large contextual inputs, for example), converts each to their corresponding vectors in a trained model, and computes a new vector as the element-wise average of the two input vectors. Here, each of the inputs is separately encoded or used to generate an encoding or vector that is then used for calculating the conceptual midpoint; such an embodiment is referred to as a direct translation implementation. This new vector (referred to as the “commonality vector”) is a representation of the “conceptual midpoint” or “contextual center” between the two input terms. The function then computes the cosine similarity between this new vector and the vectors of all other words in the vocabulary, filters out stop words and the input terms themselves, and sorts the remaining words in descending order of their cosine similarity to the new vector. In other embodiments, the commonality vector is used directly as input into a next iteration (recursion) of the commonality function and, in suchsubsequent iterations, such inputs need not necessarily be reconverted using the trained model (e.g., Word2Vec). However, in the present implementation, the most similarity function then selects a random word from the top N most similar words, with the added constraint that this word must not be too similar to either of the input terms (based on a predefined similarity threshold). If such a word cannot be found after a number of retries, the function returns the last candidate word. This word can be considered as a kind of “abstract commonality” between the two input terms.

[0064] LLMs are a more recent development in NLP and provide a different approach to NLU text compared to wordembedding techniques, such as Word2Vec. Instead of representing words as static vectors, LLMs generate word representations dynamically based on the context, making them better at handling complex language constructs and semantic relationships. With respect to the word embedding implementation above, instead of using Word2Vec embeddings and cosine similarity to find the abstract commonality between two concepts, an LLM in a prompt-based manner is used, such as where the LLM is given the prompt “Return the common theme between INPUT_1 and INPUT_2?” and have the model generate a response. As another example, the target generation LLM is GPT-3.5 and a pipeline similar to that used by ChatGPT may be used. For example, a dynamic prompt is generated and passed into the target generation LLM in order to receive an output; as an example, the following prompt is used: “return the topic most associated with the following two topics: TOPIC_1, TOPIC_2”. The result or output of the target generation LLM is a topic that may then be used as input in a next iteration.

[0065] Compared with the direct translation approach discussed above, this LLM-based approach leverages the LLM’sability to use its latent or underlying knowledge to better understand context thereby allowing it to generate coherent and contextually relevant text, which is especially important where input centers or intermediate targets are aligned to be intermediate objectives or propositions, rather than a simple general topic or broad idea. Further, an advantage of using LLMs is that they are able to capture complex semantic relationships and a wider range of commonalities, including more abstract or indirect ones. However, LLMs are generally more resource-intensive compared to word embedding techniques and their outputs may be harder to control and / or predict.

[0066] According to embodiments, the autoregressive target generation network is implemented through a LLM thattakes, as input, a prompt having the two input terms and then generates an output having the commonality vector or other representation of the conceptual midpoint. Through prompt engineering and / or fine-tuning, pretrained LLMs, such as GPT-3.5 and GPT-4, are usable for understanding and providing a response that seeks to find the commonality (represented by a commonality vector).

[0067] In either case, the recursor may be implemented as a function then uses thefunction to iteratively find words that are semantically in between two given inputs, essentially building a “semantic path” from one input to the other.

[0068] In general, the autoregressive target generation network is used to implement a kind of “semantic interpolation”between two terms or topics, finding common or intermediary topics that represent intermediate (or common) concepts between the two inputs.

[0069] Automated Conversational Steering. According to embodiments, an automated conversational steeringtechnique is used, which includes harnessing the above-described autoregressive target generation technique to steer conversations with a user in a natural manner. For example, when a human would like to talk about another topic than the one at hand (or otherwise adjust the present conversation trajectory), the human may do so in a soft or harsh manner, or somewhere in between. According to embodiments, in order to improve upon a user’s experience with a conversational text generator, such as a chatbot, the system and method provided herein perform automatic steering of conversations through introduction of intermediate targets, such as where the intermediate targets are generated based on a source and a target as input, where the source is based on the user message (conversation history) and other contextual input at the time of receiving the user message and the target is the end topic or concept to which the conversation is to be directed. Through dynamically generating prompts incorporating an intermediate target, a response message of an LLM may be controlled to introduce, discuss, or otherwise consider the intermediate target as a part of the response message.

[0070] The steering technique interacts with the personality and so the way and manner (harsh / soft) in which they aresteered is influenced by the tone-conditioning.

[0071] In embodiments, the conversational steering is tailored to the user based on the user knowledge data and / oruser-configurable options, such as one or more user options / settings that is modifiable by the user using a graphical user interface (GUI) presented on the user’s device (e.g., smartphone).

[0072] In embodiments, the user knowledge data is represented by constructing a knowledge graph. This user-specific knowledge graph, which may be specific to the user, is useful in learning about a user so as to respond in a way that cultivates an enriched interaction and conversation. Suppose, for example, during a conversation, a user mentions, “I really love mint chocolate chip ice cream” or “mmm that chocolate chip mint from Penguinbutter,” the knowledge extractor is used to identify “mint chocolate chip ice cream” as an entity, and determine that the user’s sentiment towards it is positive. In the latter one of the previous examples, it may not be clear at first what “chocolate chip mint” is referring to; however, for example, an NER technique may be used to recognize Penguinbutter as an entity and / or using latent knowledge or real-time knowledge to determine “Penguinbutter” refers to an ice cream seller. This extracted piece of information — the user’s preference for mint chocolate chip ice cream — becomes a part of the user’s evolving profile within the knowledge graph. This might be represented as a node for the user, a node for mint chocolate chip ice cream, and a positively weighted edge between them representing likes or preference points. By continually updating the knowledge graph in this way based on the user’s incoming messages, the chatbot can gradually build a more comprehensive and nuanced understanding of the user’s preferences. Thus, in future interactions, the chatbot (or message generation network or system as a whole) might leverage this information to make relevant suggestions or generate conversation, such as recommending a new dessert place that serves excellent mint chocolate chip ice cream. This continuous learning and adaptation to the user’s evolving profile via the knowledge graph amplifies the potential for personalized and engaging conversational experiences.

[0073] Knowledge Graph Inference based on Intermediate Target Extraction. In some embodiments, the intermediateextraction techniques are used to interpolate user knowledge data so as to fill-in a user’s knowledge graph. For example, a user expressing an interest in two topics, such as a common topic between the two topics, such as through use of the semantic denominator extractor. For example, say that in separate conversations, a user expressed a keen interest in both “meditation” and “aquatic life.” These two topics might seem unrelated, but they could lead us to a third topic of potential interest for the user through intermediate target extraction (e.g., using the semantic denominator extractor), where the third topic is, for example: “aquarium therapy” or “aquarium relaxation.” Aquarium therapy is the idea of using aquariums and the viewing of aquatic life as a form of relaxation and stress relief. It’s said that watching fish move through the water has a calming effect, which aligns with the principles of meditation. So, if a user has expressed interest in both meditation and aquatic life, they might be interested in this concept of aquarium therapy, where their two seemingly unrelated interests intersect. Thus, in a chatbot conversation, recognizing these two distinct interests could enable the generation of a tailored suggestion. For instance, the chatbot might mention, “Since you’re interested in both meditation and aquatic life, you might enjoy exploring the concept of aquarium therapy. It combines the tranquility of watching aquatic life with the relaxation benefits of meditation.” Moreover, this third topic could automatically be placed into the user’s knowledge graph (as a user knowledge data entry in the user knowledge data store) and may be marked with a level of confidence, an indicator that it is an inferred (rather than extracted) knowledge item / entry, and / or other information, such as feedback inferred from user’s responses to the third topic.

[0074] Low-rank Model Shared Training. This includes two training phases: a shared training phase and anindividualized training phase. In the shared training phase, the fact that the weights of the base model are frozen is exploited when having to train numerous low-rank adapters that may share common inputs in each’s respective training data set. Using low-rank adaptation for the low-rank models, the shared training phase utilizes the common-input training dataset. This allows for a single forward pass through the LLM for each shared prompt, generating an output that serves as a consistent input for each of the plurality of low-rank models. An individualized training phase may proceed, where the low-rank model is trained using the varying- input training data, such as where inputs in the respective training data sets differ. Each low-rank model is individually trained on its respective unique prompts during this phase as the prompts in the varying-input training dataset vary. The main advantage of executing the shared training phase one lies in its computational efficiency. Despite the varied outputs for different training datasets, the pre-trained LLM, a significantly large and complex model, only necessitates a single forward pass for each common input prompt. This single-pass mechanism eliminates the need for multiple forward operations through the LLM, significantly reducing computational demands and enhancing the overall training efficiency, all while preserving the consistency of responses for common prompts across the multiple low-rank models.

[0075] Speech-adapted LLM Message Generation. A speech-adapted LLM message generation system and methodare provided, including using a speech-to-text adapter that generates a model input based on audio, such as spoken words from a user, such as on a telephone or other live audio call. According to embodiments, thereis provided a method of generating a response message using a large language model (LLM), wherein the method comprises: obtaining a user audio message for a conversation session; obtaining conversation history data for the conversation session; generating an inference model input based on the user audio message and the conversation history data; and generating a response message based on an inference output that is generated by inputting the inference model input into a low-rank model and a base model, with the inference model being used as input into the low-rank model and the base model. In embodiments, at least part of the user audio message is tokenized into words conveying a user message. At least part of the user audio message is tokenized into one or more audible feature tokens, and each of the one or more audible feature tokens is a token that is defined for an audio feature. Furthermore, in embodiments, each of the audible feature tokens is determined using a classifier that classifies portions of the user audio message. One or more of the audible feature tokens indicates a tone of the user audio message and / or an emotion conveyed by the user audio message. Yet, even further, in embodiments, at least part of the user audio message is tokenized into words conveying a user message, with at least part of the user audio message being tokenized into one or more audible feature tokens. Each of the one or more audible feature tokens is a token that is defined for an audio feature.

[0076] Audio feature-based transformer. According to embodiments, there is provided a method and system foroperating a neural network, such as a LLM. The method includes (and the system is configured to) train a transformer language model using an input embedding having at least one audio feature token; obtaining a user audio message for a conversation session; generating an inference model input based on the user audio message, wherein the inference model input includes an audio feature; and generating a response message based on an inference output that is generated by inputting the inference model input into the trained transformer language model. In embodiments, positional encoding data is generated for the at least one audio feature token, and the positional encoding data specifies a location at which the audio feature token pertains. Furthermore, in embodiments, the audio feature-based transformer is used to, not only consume audio feature tokens as input, but to generate audio feature tokens as a part of the output, such as in addition to textual outputs, which may then be mapped to appropriate audio outputs, such as through use of OpenAI’s Whisper technology, for example.

[0077] With reference now to FIG.1, there is shown a LLM message communication system 10 having a LLM messagegeneration system 12 and a user (or client) device (referred herein as a “user device”) 14. Generally, the LLM message communication system 10 includes various other components for purposes of establishing a communication channel between the LLM message generation system 12 and the user device 14. The LLM message generation system 12 is used to generate a response message in response to a user message, such as to carry out a conversation, for example. In embodiments, the LLM message generation system 12 is configured as a chatbot that programmatically generates the response message based on the user message.

[0078] The user device 14 is a device used by the user in order to interface with the LLM message generation system12. The user device 14 may be any type of client device that is used by a user to interface with the LLM message generation system 12. Examples of user devices include smartphones, tablets, laptops, desktopcomputers, and other similar or like devices. In embodiments, the user device 14 communicates with the LLM message generation system 12 over an interconnected data network, such as the internet. In embodiments, the user communicates with the message generation system 12 via Short Message Service (SMS) messaging or other telecommunications-based technology. The user device 14 includes a human-machine interface (HMI), such as a display screen, audio speaker, and / or other inputs capable of conveying a message to the user.

[0079] The LLM message generation system 12 includes an input builder 116 and a message generation network118. The input builder 116 is used to generate an input that is passed into the message generation network 118, and this input is referred to as the “network input”. The message generation network 118 generates a response message based on the network input. The response message is then sent to the user device 14. One or more future iterations of this process may be performed so long as the user provides a user message, for example.

[0080] The message generation network uses an LLM to generate the response message based on the network inputand, more particularly, the LLM (or the “message generation LLM”) is used as a part of a text-to-text LLM pipeline 120 that includes the message generation network 118. In at least some embodiments, the text-to- text LLM pipeline 120 is a conversational LLM pipeline. For example, the message generation network may be implemented using OpenAI’s ChatCompletions or Completions endpoint.

[0081] The LLM message generation system 12 includes a content glossary network 104, a content retrieval APIsubsystem 106 having a content retrieval API controller 107, a glossary vector data store 108 for storing content glossary data, a content data store 110 for storing content data, an input builder 116 for generating an input for a language model, and a message generation network 118 for generating a response message based on the input using a LLM 119, in the present embodiment and implementing a text-to-text LLM pipeline 120.

[0082] Beyond the LLM message generation system 12 and its components, the LLM message communicationsystem 10 further includes an interconnected data network 114, which may be used by one or more parties 113 (which may include third parties) to provide content into the LLM message generation system 12 for various uses, namely in supplementing user messages input in a language model as a part of a message generation pipeline.

[0083] The content glossary network 104 is used to generate content glossary data for content input into the network104 and, in the present embodiment, the content glossary network 104 is implemented using an LLM, such as the LLM 119. In one embodiment, the content includes unstructured text that is too long for input into the LLM. Therefore, in such embodiments, the content is chunked or split into smaller pieces that, when concatenated, form the whole. The unstructured text data chunks may each be referred to unstructured data text entries. In some embodiments, the unstructured text data is split based on sentence, paragraph, section, chapter, column, line number, some combination thereof (e.g., column and line number, page and line number), or some other division suited for the unstructured text data. In embodiments, the LLM 119 is used to determine the type of document and, thus, a suitable structure for parsing and splitting the document or may otherwise be used to simply determine a suitable structure without necessarily determining or identifying the type of document. Inembodiments, for example, a newline character is used to split the unstructured text into paragraphs, and each paragraph is assigned an identifier (a “paragraph identifier”). The content may be split and passed to the content glossary network 104 in portions (each having one or more paragraphs, for example, but being less than the input maximum size supported by the LLM), and content glossary data generated for each input or each pass through the network 104. The content glossary data is then stored in the glossary vector data store 108.

[0084] The glossary vector data store 108 is an electronic data store that stores vector data, namely embeddingsgenerated by machine learning models. An electronic data store is a device or set of devices used for storing data electronically, such as in a database or other computer-based data store, for example, a cloud database or vector database. The glossary vector data store 108 may be implemented using a vector database platform, such as Weaviate™ or Pinecone™, for example. The glossary vector data store 108 is a vector database that stores objects that are defined according to a predefined schema, which specifies attributes of the object and which may specify other information about the attributes or object in general, such as whether the object supports multiple tenancy (i.e., partitioning / sequestering data of a single database / cluster based on tenant). The glossary vector data store 108 may be accessed via a supported API by the vector data store provider, such as through accessing Weaviate’s API using its Python API, for example. The vector data stored in the glossary vector data store 108 may be vectorized objects, which may be customized in some embodiments for the particular type of data being stored. For example, Weaviate™ enables a user to define custom schemas for objects that are vectorized and stored in its vector data store, and these objects generally are structure as an object with attribute names and associated values, where the values may be strings, integers, other numerical representations (float, double), arrays / lists, Boolean values, etc.

[0085] The content data store 110 is likewise an electronic data store that stores content. In embodiments, the contentdata store 110 may be a vector data store that stores embeddings of content generated by a LLM or other ML model. In other embodiments, the content data store 110 is a relational or no-SQL database, such as DynamoDB™. The content data stored by the content data store 110 may be received from the content retrieval API subsystem 106, which may receive the content from one or more third party data sources, such as one or more internet websites or remote computer networks, as illustrated by the one or more parties 113 in FIG.1.

[0086] The content retrieval API subsystem 106 is used to obtain content from one or more data sources, such asfrom the parties 113 via the interconnected data network 114. In embodiments, the content retrieval API subsystem 106 may directly provide the content data to the content glossary network 104. However, in other embodiments, the content glossary network 104 may be used to generate content glossary data for the content at a later time, such as a scheduled time or upon user request. However, in the illustrated embodiment of FIG. 1, the content data is passed to the content glossary network 104 so that content glossary data may be stored in the vector data store 108. The content retrieval API subsystem 106 specifies one or more endpoints, as well as parameter information, for use in calling the endpoints, where each endpoint has a particular predefined address and function. The content retrieval API subsystem 106 may be used by one or more users for causingcontent to be retrieved from one or more data sources, such as one or more third party internet sites, or for uploading content directly from the user, such as a file upload that is performed using a HTTP form submission and a Python Flask server. In embodiments, technical documents may be uploaded by one or more users to the content retrieval API subsystem 106 and content glossary data for said documents generated and stored in the glossary vector data store 108.

[0087] With reference to FIGS.2 to 4, there are shown embodiments of a LLM message generation system 100 (FIG.2), 200 (FIG.3A), 200' (FIG.3B), and 201' (FIG.4), each of which generally corresponds to the LLM message generation system 112 described above. FIGS.2 to 4 depict exemplary logical system architectures that use various components, many of which are present in the exemplary illustrated embodiments.

[0088] With reference specifically to FIG.2, there is shown an LLM message generation system 100 that includes atopic extractor network 102, a knowledge extractor network 104, a knowledge injector 109, a topic-subject application programming interface (API) subsystem 106, a user knowledge data store 108, a topic-subject data store 110, an interconnected data network 112, and a plurality of third parties 113. The LLM message generation system 100 further includes an input builder 116 and a message generation network 118 that receives input from the input builder 116.

[0089] The topic extractor network 102 is used to extract topics from the user message. In embodiments, the topicextractor network 102 uses an LLM to extract one or more topics from the user message. The topic extractor network 102 is a part of a topic extractor pipeline 103. The topic extractor pipeline 103 is used to extract topics from a user message in a text conversation, and this pipeline 103 includes pre-processing, tokenization, encoding, and inference steps. In the pre-processing step, the user message may be cleaned by removing any special characters or punctuation marks. Then, in the tokenization step, the cleaned user message is tokenized into individual tokens (e.g., words, characters, sub-tokens, audio tokens) and converted into a numerical representation using a pre-trained transformer-based LLM, such as bidirectional encoder representations from transformers (BERT) or generative pretrained transformer (GPT) LLMs. The encoded message (or latent feature message encoding) is then fed into an inference model that utilizes unsupervised learning techniques, such as clustering or topic modeling, to identify the main (or top N number of) topics of the message. The output of the pipeline may be a single topic or a list of topics that are most relevant to the user message, ranked by their probability of occurrence or other ranking or score indicating its relevancy. This approach has been shown to be effective in identifying latent topics, correlations, or other features characterizing unstructured text data.

[0090] In the previously-discussed pipeline, an LLM, such as GPT or BERT, may be used to encode the user message(which may be a text message) into a high-dimensional vector representation that captures the semantic and syntactic information of the user message. This encoding can then be used as input into the clustering or topic modeling algorithms to identify patterns and relationships in the data. For example, in clustering, the LLM- encoded feature representations (“LLM feature data”) may be used as features in a clustering algorithm, such as k-means clustering or hierarchical clustering, in order to group similar text items together based on semantical and / or syntactical similarity. Similarly, in embodiments employing topic modeling, the LLM featuredata is used as input to a latent Dirichlet allocation (LDA) model or other topic modeling techniques to identify the underlying themes or topics in the text data.

[0091] By using large, pre-trained language models, such as recent BERT and GPT based models, to encode theuser message, the rich semantic and syntactic information learned by these models during pre-training is exploited, thereby improving the accuracy and effectiveness of the topic extraction process. In one embodiment, the topic extractor network 102 uses zero-shot segmentation to identify one or more topics within the user message without having a predetermined list or set of topics upon which it was trained.

[0092] The knowledge extractor network 104 is used to extract knowledge from the user message. For example, theknowledge extractor network 104 is used as a part of a knowledge extractor pipeline 105, which may be an LLM-based pipeline (a pipeline constructed for an LLM) that extracts information from the user message, such as a user’s favorite color, a user’s food preferences, user sentiment toward objects or ideas, etc., and stores the extracted data into user knowledge data store. In embodiments, the user knowledge data store includes data representing a knowledge graph that is constructed based on the extracted knowledge from the user message.

[0093] In embodiments, the knowledge extractor incorporates or uses the topic extractor network 102 and / or portionsof the topic extractor pipeline 103 to extract topics pertaining to the user message. The discussion of the topic extractor network 102 and the topic extractor pipeline 103 above is attributed to the knowledge extractor network 104 and the knowledge extractor pipeline 105 in that the knowledge extractor network 104 performs topic extraction as a part of its knowledge extraction process. Accordingly, in embodiments, the topic extractor network 102 may be used by the knowledge extractor network 104 to extract a topic. As shown in FIG.23, a LLM message generation system 1000 (corresponding to the system 100 of FIG.2) includes a knowledge extractor network 1004 that includes the topic extractor network 1002 (corresponding to the topic extractor network 102 of FIG.2) used as a part of a knowledge extractor pipeline 1005 that includes a knowledge data extractor 1007, which is a network that takes the user message and extracted topic(s) from the topic extractor network 1002. A “network” may include one or more neural networks each of which may be represented by a matrix or tensor, also referred to as a “model”.

[0094] With reference back to FIG.2, in embodiments, the topic extractor network 102 and the knowledge extractornetwork 104 are separate and do not include a shared topic extractor model. The knowledge extractor pipeline 105 further includes entity and relational extraction techniques to identify subtopics and relationships between topics and / or entities, which are then used to update the user’s knowledge graph. For example, suppose user message includes or indicates information about a particular movie. The topic extractor is used to extract the topic of “movies” from the user message, and then to apply entity extraction to identify the specific movie mentioned in the user message. The user’s knowledge graph is updated with information about the movie, such as its plot, cast, release date, and reviews. This information can be used to provide more personalized recommendations to the user in the future. Overall, the combination of topic extraction and knowledge graph updating can help provide more relevant and accurate information to users in conversational settings.

[0095] In embodiments, the knowledge graph inference based on intermediate target extraction is used. In oneembodiment, the knowledge extractor uses zero-shot segmentation to identify one or more topics within the user message. According to embodiments, use of zero- or few-shot segmentation to extract topics and subtopics, and then to further classify the particular topic, which may be extracted using name entity recognition (NER) techniques, for example. User sentiment data is generated for the extracted topic or entity, and this data may be binary in nature (e.g., positive (good) or negative (bad)) or may be highly-dimensional, for example, stored as a latent contextual feature vector generated based on the context in which the topic or entity was referred to or evoked as a part of the user message and / or response(s) thereto.

[0096] To discern the meaning or context of a user’s message regarding (or of a user’s underlying feelings or emotionstoward) a particular topic, more comprehensive natural language processing (NLP) may be employed. For example, sentiment analysis is used to ascertain whether the user’s emotional feelings toward the topic is positive, negative, or neutral, thereby providing a more nuanced understanding of their perception of the topic. These techniques may be used to extract valuable information from the user message and update the knowledge graph with relevant information.

[0097] The extracted knowledge is stored as user knowledge data in a user knowledge data store 108, and the userknowledge data may be logically organized as a knowledge graph, at least in embodiments.

[0098] The knowledge injector 109 refers to a component that is used by the input builder 116 to query the userknowledge data store 108 for purposes of retrieving relevant user information provided the user and topic. The extracted knowledge (shown as “user data” in FIG.2 and referred to also as “user knowledge data”) is then usable by the input builder 116 for forming the input prompt that is passed into the message generation network 118.

[0099] The topic-subject API subsystem 106 is used to provide an interface for the plurality of third parties 113 so thatthe plurality of third parties 113 are able to submit requests to have a particular product, service, proposition, idea, or subject (referred to as “subject”) associated with a particular topic (or multiple topics). The topic-subject API subsystem 106 includes a topic-subject API controller 107 that is used to receive requests from the third parties and to then generate or update topic-subject data in the topic-subject data store 110. For example, a topic of television may be associated with a subject of soap operas, a type of television show; as another example, where a topic is television, the subject may be Saturday Night Live™, a particular television show. As another example, a topic may be generic or specific, and the subject may identify a single object or may convey an idea or even information. For example, in one embodiment, the topic is “Michael Jordan” and the subject is a recent update regarding Michael Jordan, such as an upcoming public appearance.

[0100] The input builder 116 is used to dynamically generate a custom input (e.g., a custom prompt) that is generatedbased on the user knowledge data and / or the topic-subject data, at least according to embodiments. With continued reference to FIG.2, there is shown a subject-injected message generation process 160, which begins with receiving a user message at the topic extractor pipeline 103 and then identifying one or more topic(s) to which the user message pertains. The input builder 116 receives the user message along with the one or more topic(s) identified by the topic extractor pipeline 103. The input builder 116 is then used to querythe user knowledge data store 108 based on the extracted or identified topics and the user (who may be identified with a user id). User knowledge data pertaining to the user is returned from the user knowledge data store 108. The user knowledge is data within the user’s knowledge graph (or associated knowledge data) that is related to one or more of the extracted topic(s). For example, a user may mention the sport, basketball, in a previous user message and a user knowledge data entry indicating a positive sentiment for basketball is stored in the user knowledge data store 108, and may be recalled by the input builder 116 when it matches the one or more topics.

[0101] According to embodiments, including the depicted embodiment, the input builder 116 uses the extractedtopic(s) to obtain a subject from the topic-subject data store 110. For example, where the extracted topic is basketball, the subject returned may be Michael Jordan, a product or news item related to Michael Jordan, or other subjects associated with the topic “basketball”. The input builder 116 receives the subject from the topic- subject data store 110, and then uses the subject to construct the prompt, which is passed as input into the message generation network 118. The message generation network 118 generates the response message, which is then sent to the user device 114 and which may be presented (e.g., displayed) to the user.

[0102] With reference now to FIGS. 3A and 3B, there is shown an LLM message generation system 200,200' thatincludes a topic extractor network 202 used to implement a topic extractor pipeline 203, a knowledge extractor network 204 used to implement a knowledge extractor pipeline 205, a topic-subject application programming interface (API) subsystem 206 having a topic-subject API controller 207, a user knowledge data store 208, a topic-subject data store 210, an interconnected data network 212, a plurality of third parties 213, an input builder 216, and a message generation network 218.

[0103] Further, with reference now specifically to FIG.3A, the LLM message generation system 200 further includesa fine-tuned head selector 220 that selects a fine-tuned head from a set of fine-tuned heads for use as a part of the message generation network 218 and, particularly, as a head on that receives output from a backbone network (e.g., pretrained LLMs, such as GPT) and generates a new output based thereon. And, still further, the LLM message generation system 200 further includes a distributed fine-tuned head training subsystem 222 that includes a distributed trainer 224 that performs distributed training for training a plurality of fine-tuned head networks 226, each of which is fine-tuned for a different task. And yet, even further, the LLM message generation system 200 includes a user history data store 228, a feedback data store 230, and a feedback data generation network 232.

[0104] With reference now specifically to FIG.3B, the LLM message generation system 200' further includes a fine-tuned low-rank model selector 220' that selects a fine-tuned low-rank model from a set of fine-tuned low-rank models for use as a part of the message generation network 218' and, particularly, as a head on that receives output from a backbone network (e.g., pretrained LLMs, such as GPT) and generates a new output based thereon. And, still further, the LLM message generation system 200’ further includes a fine-tuned low-rank model training subsystem 222' that includes a low-rank model trainer 224' that performs distributed training for training a plurality of fine-tuned low-rank models 226', each of which is fine-tuned for a different task, such asfor generating responses that take on a persona of a particular character, for example, through tone- conditioning.

[0105] Various components of FIGS.3A and 3B are discussed in the context of the LLM message generation system12 (FIG.1) and the LLM message generation system 100 (FIG.2), and the discussion of those like components is hereby incorporated and attributed to those corresponding ones of LLM message generation system 200,200'. For example, the input builder 216 corresponds to the input builder 116 of the LLM message generation system 100, and the discussion of the input builder 16 and the input builder 116 are attributed to the input builder 216 to the extent such discussion is not inconsistent with the discussion of input builder 216.

[0106] Fine-tuned models (such as fine-tuned head networks or low-rank models) may be used on a large pre-trainedlanguage model (LLM) are used to adapt the pre-trained model to a specific downstream task, such as text classification, question answering, or language generation. Pretrained LLMs, such as GPT, BERT, and RoBERTa, have been trained on a massive amount of text data, which makes them capable of understanding the nuances of natural language to a great extent; nonetheless, such pre-trained models are not specifically designed for any particular task, but are trained on a diverse set of language modeling objectives, such as predicting the next word in a sentence, predicting masked words in a sentence, and / or predicting the next sentence in a document. These pretrained models may be adapted to a specific task through fine-tuning, where a task-specific head model and / or low-rank model that is / are trained on top of the pretrained LLM. The head network or head model refers to a neural network that is connected to the top of the pretrained LLM to perform a specific task. The fine-tuned head models leverage the pre-trained knowledge of the LLM and improve the performance of the model with respect to the specific task for which it is trained through use of learning task-specific features.

[0107] The fine-tuned head selector 220 is used to select one of the plurality of fine-tuned heads 226 (referred to asthe “selected fine-tuned head” or “selected head”) for use as a part of the message generation network 218. In embodiments, the fine-tuned head selector 220 selects the selected head based on user options specified by the user.

[0108] The fine-tuned low-rank model selector 220' is used to select one of the plurality of fine-tuned low-rank models226' (referred to as the “selected fine-tuned low-rank model” or “selected low-rank model”) for use as a part of the message generation network 218'. In embodiments, the fine-tuned low-rank model selector 220' selects the selected low-rank model based on user options specified by the user (or user-configurable options).

[0109] For example, in one embodiment, with reference to FIG.3A, each fine-tuned head of the plurality of fine-tunedheads 226 is a tone-conditioned, fine-tuned head model (or “tone-conditioned fine-tuned head model”) that is fine-tuned according to a tone conditioning task that conditions the tone with which the response message conveys its meaning. Likewise, for example, in one embodiment, with reference to FIG.3B, each fine-tuned low-rank models of the plurality of fine-tuned low-rank models 226' is a tone-conditioned, fine-tuned low-rank model (or “tone-conditioned fine-tuned low-rank model”) that is fine-tuned according to a tone conditioning task that conditions the tone with which the response message conveys its meaning. In embodiments, the tone conditioning tasks are akin to one another from a broad task objective standpoint in that each tone conditioningtask seems to condition a response message output according to a predefined tone. As mentioned above, tone is referred to in the natural language context, generally referring to the way with which an idea is conveyed, such as through various linguistical, syntactical, and stylistic features, including word (or token) choice, sentence structure, punctuation, emojis, and various other ways that convey an attitude, feeling, or other emotions towards the generated idea, which is generally represented (at least in a latent representation) by the output of the backbone network.

[0110] Further, in embodiments, each of the plurality of fine-tuned models 226,226' is a customized-tone-conditioned,fine-tuned model, which may be a tone-conditioned head model (see 226 in FIG.3A) or a low-rank model (see 226' in FIG.3B) that is fine-tuned according to a customized tone conditioning task, which is a tone conditioning task that is specified at least partly in terms of custom input from a user. For example, users may provide custom user data to the system 10 via a customized training head API subsystem.

[0111] According to embodiments, as shown in FIG. 3A, the plurality of fine-tuned heads 226 are trained using thedistributed trainer 224 through performing distributed training for training a plurality of fine-tuned head networks 226. The distributed trainer 224 is a computer subsystem that is used to perform training, which generally is performed using one or more graphical processing units (GPUs).

[0112] According to embodiments, as shown in FIG.3B, the plurality of fine-tuned low-rank models 226' are trainedusing the low-rank model trainer 224' through performing training for training a plurality of fine-tuned low-rank models 226', such as using the techniques described below. For example, a shared low-rank model training technique, which is described below, is used for training the plurality fine-tuned low-rank models 226'. The low- rank model trainer 224' is a computer subsystem that is used to perform training, which generally is performed using one or more graphical processing units (GPUs). As used herein, a GPU covers any of a variety of specialized GPUs and like specialized processors such as various neural network ASICs, including tensor processing units (TPUs).

[0113] In the present embodiment, there is provided an integrated system for enhancing reinforcement learningdatasets and, more particularly, a system employing an amalgamation of heuristic-based implicit feedback techniques, explicit feedback approaches, and a specialized critic model (as used in an actor-critic model). In one embodiment, the implicit feedback techniques are used to derive meaning from a user’s interaction with the chatbot or other components of the system 200, such as deriving feedback from, for example, chat interaction duration, consecutive user return days, and user engagement with social follow widgets (e.g., Facebook™, Twitter™, LinkedIn™). Such techniques of deriving implicit sentiment or other feedback from the user may be used to determine an implicit feedback score. In addition to (or in lieu of) implicit feedback, explicit feedback may be obtained from users via binary-like approval ratings, such as, for example, represented through thumbs-up or thumbs-down signals. Other types of explicit feedback may be used as well, such as scores input by a user, audible speech-provided feedback, etc. The feedback data, which may be comprised of implicit feedback data and / or explicit feedback data, is stored in the feedback data store 230. The feedback data store 230. The feedback data of the feedback data store 230 is shown as being provided in the form of implicit feedback data from the feedback data generation network 232, which is used to generate the implicitfeedback data based on user data, such as previous user messages from the user history data store 228, user knowledge data from the user knowledge data store 208, and / or other data related to and / or from the user.

[0114] In yet another embodiment, the system 200 incorporates a fine-tuned critic model within an actor-critic modeland framework, designed to cater to a specific task, such as a tone-conditioning task, for example. Generally, in the actor-critic framework, as it relates the broader context of reinforcement learning (RL), is a framework that sets forth a set of algorithms that combine the benefits of both value-based and policy-based methods, aiming to leverage the strengths of both approaches while compensating for their respective weaknesses. At its core, this framework consists of two primary components: the actor and the critic, both of which are typically implemented as parametric functions, often in the form of neural networks. Generally, the actor in the actor- critic framework is responsible for determining the policy, which dictates the actions that the agent should take given a specific state of the environment and which is generally represented as a probability distribution over actions, either in discrete or continuous action spaces, wherefrom actions are then sampled. On the other hand, the critic evaluates the policy defined by the actor by computing the value function of the states or state- action pairs, assessing the quality of the actions chosen by the actor and providing a signal to update policy parameters.

[0115] According to embodiments, feedback data is used to drive configuration of a critic model that is directed towardevaluating the policy based on using the feedback to inform whether a particular tone or voice has been achieved. Incorporating human feedback into an RL system allows the model to learn from the expertise of a human demonstrator, reducing the amount of trial-and-error exploration required and being particularly useful in scenarios where a customized critic geared toward assessing human-like (or -defined) attributes is sought. Further, such RL based on human feedback (RLHF) techniques help in shaping the policy in a way that aligns better with human values and expectations, which is particularly important where the task set for the critic is specialized to human language tone conditioning.

[0116] To incorporate human feedback into an actor-critic framework, demonstration and intervention techniques maybe used. Collecting demonstrations includes a process where, initially, the (human) demonstrator provides demonstrations of a desired behavior. These demonstrations are used to initialize the actor’s policy. This initialization provides a decent starting point and orients the learning towards the desired goals. The (human) demonstrator observes the agent’s behavior and provides feedback. This feedback can be in the form of ratings, rankings, or even direct corrections, such as thumb’s up or thumb’s down indicators. The feedback data is used to update the critic’s value estimates, which in turn influences the actor’s policy updates. The actor updates its policy parameters based on the feedback from the specialized critic, with the goal of improving the policy to better align with the feedback. This loop of interaction, evaluation, intervention, and improvement continues until the policy reaches a desired or threshold level of performance or has meet certain threshold requirements for validation. This approach allows the system to effectively leverage human insights and judgments, leading to more efficient learning and better alignment with human understanding of tone conditioning, among other things that the disclosed framework may be applied to. Thus, the actor-critic framework, when combined with human feedback in the particular manner described herein, provides apowerful tool for reinforcement learning by combining the strengths of both value-based and policy-based methods, while also leveraging the expertise and insights of a human demonstrator; moreover, according to embodiments, employing an automatic, closed-loop, implicit-feedback RLHF framework and system.

[0117] According to embodiments, the system 200,200' includes a critic model that is distinctive from more genericcritic networks in the RLHF context, where here the specialized critic is configured to evaluate the output based on its adherence to a specified tone (“staying in character”), which is an example of tone-conditioning. In some embodiments, the specialized critic is used in the conversation target context, where the specialized critic is configured to assess the actor’s effectiveness in subtly shifting the conversation towards a predetermined topic or target.

[0118] The system can further include a conversation dataset, labeled with the composite implicit feedback, explicitfeedback, and critic scores, and this dataset may be stored in a suitable data store. This dataset is then utilized to train the head models and / or low-rank models in a multi-task learning setting, where each task is a tone- conditioning task, but where the tone is conditioned in a unique way for each different head and / or low-rank model, for example.

[0119] In an additional embodiment, the critic models themselves undergo fine-tuning to improve their evaluationaccuracy of the actor models. Each head is specifically designed to cater to a tone-task, and this fine-tuning process leverages human-labeled or -annotated data. In embodiments, a modular approach is proposed for the critic models, where each critic is specialized and interchangeable, each uniquely designed to critique a specific tone generation and used for training of a respective head.

[0120] According to embodiments, each task is directed to a particular voice or tone and is treated as a mutli-taskproblem. Loss that is attributed to each task and mixing hyperparameter (alpha) (coefficient), and this coefficient is multiplied by loss. If the tone is more generic, the coefficient is lower because it does not have to pay more attention to it. According to embodiments, each user may be treated as a separate task.

[0121] Transfer learning is used to harness the power of the pretrained LLM, but while being directed to solve aparticular task. With reference back specifically to FIG.3A, in embodiments, a distributed multi-headed training framework is provided and implemented by the distributed fine-tuned head training subsystem 222. The distributed fine-tuned head training subsystem 222 enables simultaneously training of a shared backbone network and multiple head networks in a distributed multi-task learning framework, while maintaining high levels of performance, scalability, and resource efficiency.

[0122] The distributed fine-tuned head training subsystem 222 is a distributed fine-tuned NLP head training subsystemthat is configured to train a LLM (as the backbone or body network) and a plurality of head networks, each of which corresponds to a NLP-specific task; herein, embodiments are disclosed wherein the plurality of head networks each correspond to tone-conditioning, although such a subsystem may be extended to a variety of other NLP tasks that share or have similar feature extractions, such as where the substance of the overall reply, or the context of the user input, (or somewhere in between) is similar.

[0123] According to embodiments, the distributed fine-tuned head training subsystem 222 is modeled a as distributedtraining approach applied to autonomous driving systems in connection with perception and decision-makingtasks that were to be performed simultaneously. This approach to training deep learning models is designed to simultaneously optimize a backbone, or body, network and multiple head networks at once. According to embodiments, the backbone network (corresponding to the message generation network 218) is a general- purpose feature extractor that learns a robust representation of the user message or input data, while the head networks are task-specific networks, each tailored to solve a specific task. In embodiments, the distributed fine-tuned NLP head training subsystem leverages a shared computational graph and a synchronized weight update strategy across multiple GPUs. In this setup, the input data is distributed across GPU nodes, each processing a mini-batch of samples. The forward pass begins with the backbone network, which extracts features from the input data. These features are then passed to the appropriate head networks for task-specific classification. The error signals from the head networks are backpropagated through the shared backbone, allowing for joint optimization of the entire system. To reduce communication overhead, asynchronous gradient descent is used where each GPU node independently computes gradients and updates the model weights without waiting for synchronization with other nodes. In some embodiments, the ZeRO (Zero Redundancy Optimizer) (e.g., DeepSpeed ZeRO-3) is used for distributed training optimization. According to embodiments, this results in efficient parallelization and improved convergence speed, enabling effective training of a shared backbone network and a plurality of head networks simultaneously while maintaining high levels of performance and scalability.

[0124] The distributed fine-tuned NLP head training subsystem uses an interconnected structure and coordinatedupdate mechanism employed to train the backbone and head networks simultaneously.

[0125] According to embodiments, a computational graph is used and, specifically, a directed acyclic graph (DAG),that represents the sequence of operations and data flow involved in a neural network during both the forward and backward passes. In the distributed fine-tuned NLP head training subsystem, the computational graph is shared between the backbone network and the multiple head networks. This shared computational graph allows for efficient use of resources and seamless propagation of information between the networks.

[0126] At least in embodiments, the backbone network acts as a common feature extractor, generating high-levelrepresentations of the input data, which are then passed to the head networks for task-specific classification. Consequently, the backbone and head networks are connected within the same computational graph, enabling the gradients computed in the head networks to flow back into the backbone during the backpropagation process. This facilitates the simultaneous training and optimization of both the backbone and head networks.

[0127] A synchronized weight update strategy is employed to ensure that the gradient updates across multiple GPUsremain coordinated during the training process. In this setup, each GPU node processes a mini-batch of samples and computes the gradients for the shared model. To prevent potential divergence or conflicts in weight updates, a synchronization mechanism, such as all-reduce or parameter server-based communication, is employed to aggregate the gradients from all GPU nodes.

[0128] Once the gradients are aggregated, the updated gradients are then used to perform a weight update step usingan optimization algorithm, such as Stochastic Gradient Descent (SGD) or Adam. This ensures that the modelweights remain consistent across all GPU nodes, allowing the system to maintain stability and converge to a unified solution.

[0129] In summary, the shared computational graph and synchronized weight update strategy in the distributed fine-tuned NLP head training subsystem enable efficient joint optimization of the backbone network and multiple head networks, while utilizing multiple GPUs in a coordinated and scalable manner.

[0130] Adapting this technology, which is based on Tesla’s Hydranet™, to NLP-specific tasks that generate task-specific outputs based on a common or shared backbone feature representation. Unlike previous applications of such distributed technology, the present distributed fine-tuned NLP head training subsystem 222 seeks to train multiple heads that are not used simultaneously, but are rather selectable, in real-time by a user, for example. Nonetheless, the present system harnesses the distributed training of simultaneously-operating head networks (i.e., in operation, each of the heads produces an output based on a common output of the backbone network) so as to achieve on-the-fly head swapping while enabling a training architecture that effectively, efficiently, and simultaneously trains the backbone network and / or the multiple heads at once.

[0131] With reference now to FIG.4, there is shown an LLM message generation system 201', according to a secondembodiment, that includes the knowledge extractor network 104, the topic-subject application programming interface (API) subsystem 106, the user knowledge data store 108, the topic-subject data store 110, the interconnected data network 112, the plurality of third parties 113, an input builder 116', and a message generation network 118'.

[0132] Further, the LLM message generation system 201' further includes an autoregressive target generationsubsystem 300 that implements an autoregressive target generation process in which an autoregressive target generation network 302 generates an intermediate target (and, in embodiments, a series of intermediate targets) using an LLM similarity pipeline or network in an autoregressive manner, specifically for determining the or each intermediate target based on finding a prevalent commonality or semantic-similarity between the inputs, which may be expressed as topics or ideas through a word, term, phrase, sentence, symbols, or collection thereof, generally in the form of text. It will be appreciated that the input is transformable to a tensor (of varying dimensions, which may be dependent on the application) and this representation may be used as input into the network (such as where vector stores are used to store source, target, and intermediate target data. In embodiments, the autoregressive process takes in two inputs referred to as a source and a target and generates an intermediate target that is representative of a prevalent commonality or semantic-similarity between the source and the target, often corresponding to a common denominator between the source and target. In the context of finding a common denominator between two topics or ideas, “semantic similarity” refers to the degree to which the meaning of the words and concepts used in those topics or ideas overlap; moreover, it is a measure of how closely related the two concepts or ideas are in terms of meaning, rather than just their superficial similarity based on shared words or phrases. For example, “dogs” and “wolves” may each be highly associated with common words like “fur” or “pack,” but the semantic similarity between them would be higher if they also share concepts like “canine” or “medium-sized, four-legged mammal.” In this case, “canine” and “medium-sized, four-legged mammal” may achieve a higher semantic similarity score due to such topics beingmore specific than other concepts or topics, such as “fur” or “pack,” which are more generic in nature. The autoregressive target generation network 302, thus, implements a semantic similarity pipeline that determines concepts or topics that score high in terms of semantic similarity and generates an output that based thereon.

[0133] In embodiments, the autoregressive target generation network 302 uses an instruction-following LLM, such agenerative pre-trained transformer (GPT)-based LLM (e.g., GPT-3.5 or GPT-4), that is provided an input that instructs the instruction-following LLM with the following prompt: “in a concise manner, return the concept or idea that is most semantically similar to the following two inputs: (1) INPUT_1 and (2) INPUT_2”. In embodiments, one-shot or few-shot-learning pipeline is used where one or more examples are input as a part of the prompt as well.

[0134] In other embodiments, other techniques may be used to implement the autoregressive target generationnetwork 302, such as a dedicated pipeline that is formulated for such a task.

[0135] By analyzing the semantic similarity between topics or ideas, one can identify the most salient commondenominator that connects them. In one embodiment, the following pipeline is used: (1) tokenize INPUT_1 and INPUT_2 using a tokenizer; (2) generate respective embeddings for INPUT_1 and INPUT_2; (3) merge the respective embeddings, such as performing an arithmetic mean of elements along an axis; and (4) determine a most similar topic based on assessing similarity between the merged embedding and embeddings of a predefined list of vocab words, such as through use of cosine similarity. The previous four steps may then proceed in a recursive manner in which one of the two inputs is replaced with the output of the previous iteration, and generally for producing a regressive output that predicts the semantic denominator.

[0136] For example, with reference to FIG.5, there is shown a depiction of outputs generated using an autoregressivetarget generation process having a recursion level or depth set to three (3). At each level, a set of intermediate targets are generated: at level-1, a single intermediate target is generated and this intermediate target is referred to as the “source-target center”; at level-2, two intermediate targets are generated, with the first INTERMEDIATE_TARGET_2_1 being a semantic denominator between the source (or INPUT_1) and the source-target center, and with the second INTERMEDIATE_TARGET_2_2 being a semantic denominator (or contextual center) between the source-target center and the target (or INPUT_2); at level-3, four intermediate targets are generated, with the first INTERMEDIATE_TARGET_3_1 being a semantic denominator between the source and INTERMEDIATE_TARGET_2_1, with the second INTERMEDIATE_TARGET_3_2 being a semantic denominator between the INTERMEDIATE_TARGET_2_1 and the source-target center, with the third INTERMEDIATE_TARGET_3_3 being a semantic denominator between the source-target center and the INTERMEDIATE_TARGET_2_2, and with the fourth INTERMEDIATE_TARGET_3_4 being a semantic denominator between the INTERMEDIATE_TARGET_2_2 and the target. In embodiments, the autoregressive target generation network prioritizes generating intermediate targets closer in the target pathway to the source since those intermediate targets are to be used by the input builder sooner.

[0137] After the intermediate targets are generated, the intermediate targets are serialized to form a target pathway(or “serialized target pathway”) that begins at the source and ends at the target, interpolated therebetween withthe generated intermediate targets. For example, tree serialization is used to transform the tree into a serialized set or ordered list of nodes, where each node corresponds to one of the inputs or an intermediate target.

[0138] The autoregressive target generation system 300 includes the autoregressive target generation network 302and a target builder 304, which is similar in general purpose as the input builders discussed herein in that its general objective is generating a suitable input in a generation (or ML) network, but the target builder 304 is more specific in that the target builder 304 is used to facilitate the autoregressive operation of the autoregressive target generation network 302. Furthermore, the target builder 304 is used to set an initial target, which may be determined based on the user message and / or other information relating to the user, such as information from the user knowledge data store 108.

[0139] In embodiments, a target-topic data store 306 is used to store target-topic data entries, which specify aparticular target for a given subject, topic, or idea (collectively represented by the “topic”). According to embodiments, the target-topic data store 306 is queried with information based on a user’s message. In embodiments, other contextual information may be used, such as previous user messages (e.g., chat history), and / or information from the user knowledge data store 108. The target-topic data store 306 also may include target-topic data that is generated based on information received via the topic-subject API subsystem 106. In embodiments, the target-topic data store 306 is built using data from the topic-subject data store 110 and / or may be built or otherwise managed / updated using the topic-subject API subsystem 106, for example.

[0140] Further, such source data or source embedding may be supplemented with other contextual information, suchas that from the user knowledge data store 108. In embodiments, the topic extractor network 102 is introduced into the system 201' and used to extract topic data about the user message, which may be then used here to determine a target (e.g., through querying the target-topic data store 306) and / or the user knowledge data store 108.

[0141] The present system 201' and, particularly, the autoregressive target generation system 300, may be used in avariety of settings and for a variety of purposes. For example, in an “educational context” (where the system 201' is generally engaged with learning or education through a chatbot or other conversational agent), the autoregressive target generation system 300 may be used to set a target lesson objective based on the source, which takes into account present circumstances regarding use of the message generation network 118', such as a previous message generated and / or the user message. As a particular example in the educational context, the target is set to an educational objective (referred to as a “target lesson objective”) and intermediate targets are extracted using the autoregressive target generation system 300. The intermediate targets are then serialized from source until target as shown in the example in FIG.5. Thereafter, the intermediate targets may be used by the input builder 116' for purposes of being used to generate the input that is then passed into the message generation network 118'.

[0142] The Socratic method is an epistemological inquiry technique characterized by the use of methodical inquiry toprobe, elucidate, and investigate underlying assumptions, concepts, and reasoning behind a given proposition or argument. The Socratic method is often regarded as a form of inquiry that seeks to expose and challenge the underlying assumptions and presuppositions that inform a particular line of reasoning or argumentation.Through a series of carefully crafted questions and / or assertions, the Socratic interlocutor guides the conversation towards a deeper analysis of the fundamental concepts and principles at play, with the aim of clarifying and refining the original proposition or claim. The Socratic method may be used for directing the respondent or user to a particular goal or target, and this may be referred to as a goal-directed Socratic method or goal-directed learning. In embodiments where the present system 201’ is used in an educational context, the system 201’ operates as an interlocutor.

[0143] In embodiments, the input builder 116' structures the input according to an elenchus technique, which is atechnique used in the Socratic method to inquire into a proposition or argument through questioning and examining (or cross-examining) the proposition or assertion. For example: Source: The respondent believes that the best way to train for a marathon is by running as much as possible. Target Objective: To arrive at a more nuanced and comprehensive understanding of the different training methods and strategies for running a marathon. Intermediate Objectives: 1. Examine the ways in which the respondent’s previous training methods may have limitedtheir potential for improvement, and consider alternative strategies for enhancing performance. 2. Evaluate the potential benefits and drawbacks of different training methods, such as intervaltraining, cross-training, and rest and recovery, for running a marathon. 3. Synthesize the insights gained from the previous objectives to arrive at a morecomprehensive and nuanced understanding of the different training methods and strategies for running a marathon, with consideration given to the respondent’s own training goals and limitations.

[0144] Using the elenchus process, the Socratic interlocutor would first guide the respondent towards an intermediateobjective is specified as a middle ground between the user’s belief in the importance of running (the source) and the target objective of a more comprehensive understanding of training strategies. From there, the interlocutor would guide the respondent towards an intermediate objective that evaluates different training methods, and then towards a final intermediate objective that synthesizes the insights gained to arrive at a more nuanced understanding of training strategies.

[0145] The present system 201’ may be adapted to act as the interlocutor, at least in embodiments. For example, theautoregressive target generation network 302 may be configured to generate halfway learning objective data as the intermediate target or objective; this intermediate target corresponds to a convergent point between the two inputs, such as the source (or the initial assertions, statements, and other context of the user message) and the target (or the ultimate objective of attaining a more nuanced and comprehensive understanding of a particular topic). Through a systematic approach, the autoregressive interlocutor (implemented by the autoregressive target generation network 302) steers, nudges, or otherwise guides the user towards anintermediate objective that takes into account both the user’s starting point and the broader objective, allowing the respondent to explore and assess alternative perspectives.

[0146] The autoregressive target generation system 300 may be used in other contexts as well. For example, theautoregressive target generation system 300 is usable to steer the conversation toward a target objective, but in a manner that is subtle so as not to appear to be deflecting or abruptly redirecting the conversation. Moreover, the autoregressive target generation system 300 enables generation of the intermediate targets at the time of receiving the user message (a first user message) without knowing what the user’s subsequent messages in the future will say. According to embodiments, user data is obtained from the user knowledge data store 108, such as in response to querying the user knowledge data store 108 with the user and a topic. In embodiments, the topic is extracted from the user message using the topic extractor network 102, which may be incorporated into the system 201', as will be appreciated by those skilled in the art.

[0147] For example, in a conversation with the user where the upcoming weekend is being discussed, the topicextractor network 102 is used to extract topic(s), such as “weekend”, and then the topic(s) (or any one or more thereof) are used to obtain a target objective (conversation topic) that is set based on the extracted topic(s). In embodiments, predefined topic-target data is determined where the predefined topic-target data specifies a target along with a topic and, in embodiments, the target to topic relationship may be a 1-to-1, 1-to-many, or many-to-many relationship; for example, numerous different topics may be specified for a given target; moreover, in embodiments, each of (or one or more of) the different topics may be associated with more than one target.

[0148] The “weekend” is extracted as a main topic item using the topic extractor network 102 and “barbeque sauce”is obtained as the target based on using “weekend” as input for querying the topic-target data store; then, in this example, an intermediate target is obtained, such as “grilling”, as that generally takes place on a weekend and for which barbeque sauce is often used. The autoregressive target generation network may recursively iterate a certain number of times, thereby generating a plurality of intermediate targets, as discussed above. These targets may then be progressively introduced into the conversation via dynamic prompting techniques, such as where the LLM is directed to respond to the user message, but to mention the INTERMEDIATE_TARGET to the extent it is not off-putting or awkward to do so in the response.

[0149] According to embodiments, the intermediate targets are generated and used to set sub-objectives, spanningbetween the source to target and used for various purposes, such as to guide the user and / or to evaluate the user message to determine whether a sub-objective (or “intermediate objective”) has been achieved. For example, the input builder 116' modifies or otherwise generates a prompt (or other input) based on the intermediate target and this input is then passed into the message generation network 118' for purposes of generating a response message that is sent to a user device 114 (corresponding to the user device 14). In one embodiment, the prompt specifies the user message as well as intermediate target data, which is data that is generated based on the intermediate target. The intermediate target data may be, for example, instructive text instructing the message generation network 218 to respond in a particular manner based on the intermediate target; for example, the instructive text is “respond to the USER_MESSAGE while suggestingINTERMEDIATE_TARGET”. This exemplary instructive prompt may be input into the message generation network 118' along with the user message, which is represented by the USER_MESSAGE variable in the previous example. The intermediate target, which is represented by the INTERMEDIATE_TARGET variable in the previous example, may correspond to.

[0150] With reference now to FIGS. 6 and 7, there is shown, according to one embodiment, a LLM messagecommunication system 410 having a LLM message generation system 412, a user (or client) device (referred herein as a “user device”) 414, as well as an interconnected data network 402 and a plurality of custom data parties 404. Generally, the LLM message communication system 410 includes various other components for purposes of establishing a communication channel between the LLM message generation system 412 and the user device 414. The user device 414 is analogous to the user device 14 and that discussion is hereby incorporated and attributed to the user device 414 to the extent that such discussion is not inconsistent with the discussion of the user device 414. The LLM message generation system 412 is used to generate a response message in response to a user message, such as to carry out a conversation, for example. In embodiments, the LLM message generation system 412 is configured as a chatbot that programmatically generates the response message based on the user message. In embodiments, the LLM message communication system 410 may include one or more components of the previous LLM generation systems 100 (FIG. 2), 200 (FIG. 3A), 200' (FIG. 3B), and 201' (FIG. 4). Indeed, in embodiments, the LLM message generation system 412 includes components discussed above with respect to the system 200' (FIG.3B).

[0151] The LLM message generation system 412 includes a message generation subsystem 415 having an inputbuilder 416, a message generation network 418 having a base model 420 and a low-rank model 422, a network selector 424 (corresponding to the low-rank model selector 220' (FIG.3B)), a custom training subsystem 428 having a custom data training API subsystem 430 and a custom data training subsystem 440. In embodiments, the custom training subsystem 428 is used as a part of an end-to-end autonomous custom training system 426, which is discussed more below. It will be appreciated that, although the base model 420 is illustrated as having both an encoder and decoder, various pretrained LLMs of various architectures may be used depending on the task at hand; for example, in the case of message generation for conversational agents, such as ChatGPT™, the base model 420 may be a decoder-based LLM. An example of a decoder-based large language model is GPT (Generative Pretrained Transformer) developed by OpenAI™. GPT is an autoregressive model that uses context from the left of the input to predict the next word, and it learns to predict the probability of a word given the previous words in a sentence.

[0152] The custom data training subsystem 440 is used to train low-rank models using custom data where the low-rank models are to be used (as the selected low-rank model 422) during inference by the message generation network 418. The custom data training API subsystem 430 is used to obtain custom user data from a custom data party, which is any party that provides data to the custom training subsystem for purposes of using the provided data (referred to as “custom user data”) for purposes of training. Although the exemplary embodiment of depicts three custom data parties 404, although any number of custom data parties may use the custom training API subsystem 430, according to embodiments.

[0153] In embodiments, the custom training subsystem 428 is used as a part of the end-to-end autonomous customtraining system 426. The end-to-end autonomous custom training system 426 is characterizable herein as an “end-to-end” system in that the system 426 provides a system that begins, at a starting end, with custom user data received and ends, at another end, with a trained model having been trained using the custom user data; and, the system 426 is characterizable herein as an “end-to-end autonomous” system in that the system 426 is an end-to-end system that is performed for autonomously, meaning without manual user intervention. According to the depicted embodiment, the end-to-end autonomous custom training system 426 is implemented using the custom data training API subsystem 430 and the custom data training subsystem 440.

[0154] The custom training API subsystem 430 includes a custom training API controller 434 and a custom user datastore 436 that is used to store the custom user data received by the custom training API subsystem 430. In embodiments, the custom training API controller 434 implements a custom training API that is an API for providing custom user data to the custom training API subsystem 430. The custom user data store 436 is a data store that is used to store custom user data, such as the custom user data received via the custom training API subsystem 430.

[0155] In embodiments, the custom user data store 436 includes conversation history for a user. The conversationhistory may be a text message history (e.g., SMS history) for the user, a transcript in which the user is a participant, and / or may be other content, such as audio or other text content, generated by the user. In embodiments, user audio received from the user, such as from the user device 414, may be converted to text using ASR and one or more audio features extracted, and that data may be stored in the user knowledge data store. According to embodiments, the user knowledge data for a user is used as training data for training a low-rank adapter or other low-rank model so that a message generation network incorporating the low-rank model. For example, in one embodiment, the low-rank model is trained on the conversation history and, then, the message generation network is configured to generate, as a response message, a message that seems as though the message was spoken or constructed by the user. This may involve, for example, setting stop words (StoppingCriteria) for inference generation when specifying generation configuration settings for inference. According to embodiments, the user is able to use a custom training API subsystem (e.g., the custom training API subsystem 430) to upload conversation history data in order to have a custom low-rank model generated that is adapted to produce text output trained specifically for the user—this model is referred to herein as a “me model”. Further, according to embodiments, the me model is used in the message generation network 418 for generating messages that sound like the user (“me”).

[0156] The custom data training subsystem 440 includes a training data customizer 442, a low-rank model trainer 444(corresponding to the low-rank model trainer 224' (FIG.3B)), a custom low-rank model data store (or model repository) 446 storing a plurality of custom low-rank models 448, an application-specific training data store 450 having a plurality of application-specific training data (ASTD) entries 452-n, and an application-specific trained model data store 460 having application-specific trained model (ASTM) entries 462-n, where n is from1 to N and where N is the number of applications. In embodiments, the custom low-rank model data store 446 is a model registry that, for each model of the low-rank models 448, stores the model, the model version, themodel stage, and a model alias, for example. As used herein, an “application” (when not used as the “application” in “API”) refers to a predefined task-specific use for an LLM, such as providing conversational response messages that are responsive to a user message and other contextual information that is provided in the input through an application-specific structured prompt. As used herein, a “structured prompt” refers to a prompt that is structured according to a predefined prompt structure. As used herein, a “predefined prompt structure” refers to a set of predefined fields that are used for calling attention to certain aspects of a value for the predefined field when used in a prompt or other input. Fine-tuning of a pretrained LLM using training data that consistently uses various the predefined files of the predefined prompt structure results in a fine-tuned LLM that is tuned toward certain fields and their meanings as inferred via the training data set. In the case of LLMs, a prompt is oftentimes in the form of a human-readable text string that includes certain predefined prompt field identifiers and associated prompt field values. The predefined prompt field identifiers refers to a predefined string of tokens that is to be used in a consistent manner in terms of form when used as input into a message generation network. And the associated prompt field values are one or more tokens that are values for its associated predefined prompt field (as indicated by its associated predefined prompt field identifier).

[0157] Here, each application uses an application-specific trained model 468 as the base model 420 in the messagegeneration network 418 and for training the low-rank model 422, which is performed by the low-rank model trainer 444. However, in the present embodiment, prior to using the low-rank adapter training, the application- specific trained model 468 is generated through training a pretrained LLM using application-specific training data, which is training data structured according to a predefined prompt structure specific to the application. For example, in the context of LLM-based chatbots, a first application is defined for providing conversational response messages for use by a non-playing character (NPC) in a video game, where the prompt fields may be used to define various attributes about the NPC; in this same broad chatbot context, a second application is defined, for example, as providing conversational response messages for use by a business in automating conversations with customers or potential customers; and, again in this same broad chatbot, for example, a third application is defined as providing conversational response messages for use by a chatbot that answers domain-specific questions, such as those that are specially-adapted to a particular technical field or other specialty. For example, in the first application, the prompt fields may include attributes about the character (e.g., size, gender, catchphrases, sentiment (good guy / bad guy)) while the prompt fields for the second application may include attributes about the business (e.g., business hours, business processes, type of business, location of business). Of course, these are only but a few examples.

[0158] Moreover, while the previous examples provide applications that are for different kinds or types of uses, multipleapplication-specific trained models may be generated for use within any one of these areas. As an illustrative example, in the video game context, different types of video games may involve different attributes for a character. For example, consider attributes typically used in a role-playing game (RPG) versus a narrative- driven adventure game.

[0159] For characters in an RPG, attributes often include health, magic, strength, agility, intelligence, and charisma.Health is an attribute that typically represents the overall vitality or life force of the character. Magic (orsupernatural) is an attribute that generally refers to a mystical or fantastical element, such as an ability to cast spells, use magical items, or resist magical attacks. Strength is an attribute often determines the physical power of a character, influencing their ability to deal damage, lift heavy objects, or successfully perform certain physical tasks. Agility is an attribute typically affects a character’s speed, reflexes, and dodging ability, such as where a higher agility attribute might mean that the character can move quickly, avoid attacks, or perform actions at a faster rate. Intelligence is an attribute that refers to a character’s intelligence attribute affects their ability to use certain skills, decipher puzzles, or cast powerful spells, and may also affect how quickly a character can learn or understand an input, such as a user message. Charisma is an attribute that refers to how a character is able to influence other characters (e.g., NPCs) that interact with the character. A higher charisma could lead to better negotiation outcomes, more favorable reactions from NPCs, or even the ability to persuade or intimidate others.

[0160] On the other hand, characters in narrative-driven adventure or interactive fiction games, attributes may ofteninclude empathy, persuasion, insight, knowledge, wit, and charm. Empathy is an attribute that typically represents a character’s capacity to understand and respond to the emotions of non-player characters (NPCs). Persuasion is an attribute that generally determines a character’s success in convincing others to see their point of view or make certain decisions. Insight is an attribute that often influences a character’s ability to discern hidden information, lies, or the true intentions behind an NPC’s words or actions. Knowledge is an attribute that denotes a character’s understanding or memory of various subjects, which may be instrumental in providing context, solving puzzles, or engaging in deeper, more insightful conversations with NPCs. Wit is an attribute that refers to a character’s ability to respond quickly and cleverly in dialogues or situations, often leading to unique conversation paths. Lastly, Charm is an attribute that characterizes a character’s appeal or charisma, influencing how NPCs react to and interact with the character. A higher charm attribute could result in friendlier interactions, more favorable reactions from NPCs, or potential alliances.

[0161] Thus, in embodiments, a first prompt structure having attributes such as health, magic, strength, agility,intelligence, and charisma may be defined for an RPG-specific trained model where the application is for training RPG characters. Meanwhile, a second prompt structure having attributes like empathy, persuasion, insight, knowledge, wit, and charm may be defined for an interactive narrative-specific trained model, designed for developing characters within narrative-driven adventure or interactive fiction games. The first prompt structure is used to define application-specific training attribute (ASTA) data that is stored in a data store; for example, in one embodiment, the ASTA data is stored in the application-specific training data store 450 as a part of the first ASTD entry 452-1. Likewise, the second prompt structure is used to define application-specific training attribute (ASTA) data that is stored in a data store; for example, in one embodiment, this ASTA data is stored in the application-specific training data store 450 as a part of the second ASTD entry 452-2. The ASTD entries 452-n are each for an application, where each application includes the ASTA data for the application and application-specific prompt template (ASPT) data. ASPT data is a string of tokens (e.g., one or more alphanumeric and other like characters) that includes one or more predefined prompt field identifiers and, inembodiments, the ASPT data includes placeholder identifiers for indicating portions whereat the associated prompt field values should be included.

[0162] In embodiments, the ASTA data is used by the custom training API controller 434 to validate or otherwiseverify that a request (e.g., API request) from the custom data party is valid, such as ensuring that the appropriate information is provided as a part of the custom user data. In embodiments, ensuring the appropriate information is received may be facilitated by a graphical user interface that is constructed according to the ASTA data, such as through creating graphical field labels (e.g., text labels reading “Health:”) and corresponding graphical input fields (e.g., text input fields, such as an <input type=‘text’ / > html element). The GUI is dynamic in that the fields and corresponding input elements are modified based on the selected application; or, in other embodiments, a different, static GUI may be constructed and deployed for each application. According to implementations, each application may have a separately-defined API call (e.g., a separate REST path); and, in other embodiments, each application may share a single API call, where the call request is constructed based on the relevant attributes and the associated values filled out by the custom data party. The custom training API controller 434 processes the API calls, which may include a payload of custom user data to be used for training a low-rank adapter; in other embodiments, the API request may specify a location, such as a universal resource location (URL) at which the custom user data is located.

[0163] With reference to FIG.7, there is shown a quantization operation that is performed on the pretrained LLM 466so as to generate a quantized pretrained LLM 467, which is then fine-tuned or otherwise trained by the application-specific trainer. In at least some embodiments, a generative post-training quantization (GPTQ) for generative pretrained transformers technique is performed. In embodiments, quantization to a lower number of bits allows for faster inference speeds. In FIG.7, the application-specific trainer (“app-specific trainer”) 464 is used to fine-tune a pretrained LLM so as to generate an application-specific trained model or application- specific fine-tuned model.

[0164] In embodiments, the LLM message generation system 412 further includes a message generation API thatprovides an output of the message generation based on a request that is prepared according to a predefined message generation API that is used for requesting an inference output based on a user input, which oftentimes includes a message or other user message from the user of the user device 414.

[0165] In embodiments, a first custom data party seeks to obtain a custom-trained model that is trained based oncustom user data provided by the first custom data party; likewise, a second custom data party 404b seeks to obtain a custom-trained model that is trained based on custom user data provided by the second custom data party. Take, for instance, an example where the first custom data party is a first video game creator of a video game that is an RPG and that includes numerous different characters. And, in this example, the second first custom data party is a second video game creator of a video game that is an RPG and that includes numerous different characters as well.

[0166] With reference now to FIGS.12 and 13, there are shown two LLM message generation systems 412'',412''',which are each analogous to the LLM message generation system 412 and 412', and each further includes a dual-selectable model framework 413'',413''' that enables a low-rank model 422 and a selectable head model421'',421''' to be selected during runtime and used by the LLM message generation system 412'',412'''. The selected head model is shown in an active position in FIGS.12 and 13 where the head model is downloaded to a computer and ready for performing a forward pass for inference on the computer. The selected head model in this active position may be referred to as an active head model. The two LLM message generation systems 412'',412''' are generally the same, except that each’s message generation network 418'',418''' includes a different arrangement of its dual-selectable models. In particular, the LLM message generation systems 412'' implements an outer head framework in which the fine-tuned head receives a combined input that is made up of the low-rank model 422 output and the base model 420 output whereas the LLM message generation systems 412''' implements an inner head framework in which the fine-tuned head receives an input from the base model (not from the low-rank model) and then the fine-tuned head generates an output that is then combined with the low-rank model 422 output. Each of the LLM message generation system 412'',412''' includes an input builder 416'',416''', which may be a speech-to-text input builder (such as the input builder 1016 discussed below) or another text input builder (such as ones that may be used for the input builder 416). Further, the input builder 416'',416''' is used to provide data pertaining to the user message or input to the fine- tuned head selector 420'',420'''. In embodiments, the fine-tuned head selector 420'',420''' selects a fine-tuned head based on user-configurable options, such as a selected tone for the response messages. In embodiments, a tone-conditioned model that is trained toward conditioning the message in order to incorporate and use a particular tone. In such an example, a user may specify a particular tone to use (“manual tone selection”), or the system may automatically determine a tone (from a predetermined list of tones, each of which may be associated with a different tone-conditioned model, for example) to use (“automatic tone selection”). In embodiments, the fine-tune head selector 420'',420''' uses context provided by the input builder 416'',416''' and / or other data available by the system in order to perform automatic tone selection.

[0167] With reference to FIG.8, there is provided a method of autonomously custom training a low-rank model havinga feed forward neural network, such as a low-rank adapter implemented using the LoRA for LLM Technology, for use with a large language model (LLM). The method begins with step 610, wherein custom user data is received from a custom data party. For example, with reference to FIG.6, the custom user data is received at a custom training API subsystem 430 and, more particularly, at the custom training API controller 434. The custom training API controller 434 then may process the request and store the custom user data into the custom user data store 436. The method 600 continues to step 620.

[0168] In step 620, a base model is selected based on the custom user data. For example, as discussed above, apretrained LLM is generally trained to a generic task but may be fine-tuned so that it is directed toward a particular task or application. Here, a plurality of fine-tuned models are generated through fine-tuning a pretrained model using fine-tuning training data. For example, the pretrained model may correspond to a causal decoder-only model, such as Falcon-40B ("tiiuae / falcon-40b") built by Technology Innovation Institute (TII), which is trained on 1,000 billion tokens of RefinedWeb ("tiiuae / falcon-refinedweb"). For example, a chat- based fine-tuned LLM may be generated using the Falcon-40B and fine-tuning it using training data that includes conversations. For example, in some embodiments, such training data is automatically generated byhaving top-of-the-line GPT-based chat models, such as ChatGPT, converse with itself. In some embodiments, fine-tuning may be performed in multiple stages so that a single pretrained LLM is fine-tuned to generate a first- level fine-tuned model and, then, the first-level fine-tuned model is fine-tuned again (on different training data) to generate a second-level fine-tuned model. This two-stage fine-tuning seeks to generate a plurality of models that are fine-tuned for an application, but that share some common, more general task, such as text generation. The first-level fine-tuned model may correspond to an instruct model, such as "tiiuae / falcon-40b-instruct", and the second-level fine-tuned model is application-specific in that it is fine-tuned using a predefined prompt structure that is configured for a particular application, such as for RPG games, as discussed above. The custom user data may specify which type of game (e.g., RPG, narrative-driven adventure game), and the corresponding application-specific fine-tuned model is then selected. The method 600 continues to step 630.

[0169] In step 630, custom training data is generated based on the custom user data and the selected base model tobe used. In embodiments, the selected fine-tuned model is a pretrained model that is fine-tuned using a predefined prompt structure and, in at least some embodiments, the predefined prompt structure is different for each application-specific fine-tuned model.

[0170] Upon receiving custom user data through an API, the data must first be cleaned and preprocessed to ensureit is in an appropriate format for training. This might involve removing unnecessary or irrelevant information, handling missing or inconsistent data, and potentially normalizing or transforming the data if required. The cleaned data is then integrated into a predefined prompt structure to generate the training data. This integration could be a simple string substitution or may involve more complex processing depending on the prompt structure and the nature of the data. The resulting training data, consisting of the input prompts derived from the custom user data embedded in the predefined prompt structure, is then ready to be used for fine-tuning the pretrained language model.

[0171] The predefined prompt structure specifies one or more (generally, a plurality of) predefined prompt fieldidentifiers and its associated prompt field values may be filled out based on the custom user data so as to generate the custom training data. As mentioned above, the predefined prompt field identifiers refers to a predefined string of tokens that is to be used in a consistent manner in terms of form when used as input into a message generation network. And the associated prompt field values are one or more tokens that are values for its associated predefined prompt field (as indicated by its associated predefined prompt field identifier). Thus, here, the custom user data is used to populate prompt field values, each being associated with a predefined field identifier. In embodiments, the training data is supervised training data in that it includes input, output pairs. In the present embodiment, generation of supervised custom training data is performed, such as where the custom user data provides various custom information that may be used to fill out a predefined custom training template data. As used herein, “predefined custom training template data” refers to data that is predefined and used for training, even though such data may first be modified, such as to fill in placeholders with corresponding custom user data provided by the custom user data party. More specifically, at least in embodiments, the predefined custom training template data is customized according to a particular application or use-case, such as for creating a chatbot that responds to customers or potential customers about thebusiness’ particulars or for creating a chatbot that responds in a particular character’s voice, such as William Shakespeare.

[0172] In an example, where the chatbot is used as a business FAQ or help chatbot, the predefined custom trainingtemplate data (“PCTTD-1”) may include data, such as:

[0173] This PCTTD is then used to generate custom training data through replacing the placeholders with data fromthe custom user data. This predefined custom training template data can be filled in with custom user data to generate the custom training data. Once filled in, it may look like this:

[0174] As another example, where the chatbot is used as a character-based chatbot, the predefined custom trainingtemplate data (“PCTTD-2”) may include data, such as: This PCTTD is then used to generate custom training data through replacing the placeholders with data from the custom user data. This predefined custom training template data can be filled in with custom user data to generate the custom training data. Once filled in, it may look like this:

[0175] In some embodiments, the training data is not in the form of input, output pairs but includes conversationalinputs, such as text conversations between two individuals or entities. In such embodiments, the custom user data may include such text conversations, such as chat message history from one or more conversations, each of which may be between two or more users or entities.

[0176] As shown in FIG.7, there is illustrated the application-specific training data store 450 that includes a pluralityof ASTD entries 452-n as discussed above. In embodiments, there is one entry per application and the application, which may be specified by the customer user data or otherwise determined or inferred based on the custom user data, is then used to query the application-specific training data store 450 so as to obtain the ASTD, which corresponds to the predefined custom training template data discussed above. This predefined custom training template data is used to generate the custom training data, which may then be stored in a data store, such as a custom training data store (not shown). The method 600 continues to step 640.

[0177] In step 640, the selected base model is obtained for training a low-rank model, such as a low-rank adapter. Inembodiments, the low-rank model is a low-rank adapter and includes an encoder and a decoder in an autoencoder arrangement, such as defined by the LoRA for LLM Technology. The selected fine-tuned model may be trained using a transformer trainer, such as the trainer provided by HuggingFace™ transformers library (presently available at git+https: / / github.com / huggingface / transformers.git), in order to train a low-rank neural network, such as a low-rank adapter. The following provides an example of training a low-rank adapter, according to one embodiment. In embodiments, the following low-rank adapter configuration options may be used as defined by the LoraConfig library provided by “bitsandbytes”, which is a lightweight wrapper around Compute Unified Device Architecture (CUDA) custom functions, in particular 8-bit optimizers and quantization functions.

[0178] In embodiments, various quantization configuration values may be set for the application-specific fine-tunedmodel, such as whether 4-bit quantization is used (e.g., NormalFloat 4, FloatingPoint 4 quantization), whether double quantization (or other m-nested quantization, where m is the number of layers), whether the pretrained model is initially loaded as a quantized 4-bit representation, a compute data type, etc. An example below is provided with respect to the BitsAndBytesConfig library.

[0179] The BitsAndBytesConfig is a configuration object in the HuggingFace transformers library designed to facilitateparameter setup for the BitsAndBytes module. The attribute load_in_4bit=True dictates that the model shall be loaded using 4-bit precision, a setting intended to minimize the model’s memory footprint and possibly expedite training and inference based on the underlying hardware. The parameter bnb_4bit_quant_type="nf4" configures the 4-bit quantization type to "nf4", or Normalized Float 4, a specific variant of quantization known for delivering superior performance, as substantiated by theoretical and empirical evidence. The setting bnb_4bit_use_double_quant=True triggers a secondary quantization subsequent to the initial one, thereby conserving an additional 0.4 bits per parameter, further contributing to a reduction in the model’s memoryconsumption. Lastly, the bnb_4bit_compute_dtype=torch.bfloat16 parameter designates the data type employed for computations to torch.bfloat16 (Brain Floating Point 16), a 16-bit floating-point representation that proffers the benefits of reduced-precision training and inferencing without necessitating alterations to the model’s hyperparameters. In embodiments, this configuration is used to prepare the model to utilize a 4-bit precision accompanied by a second round of quantization and to employ bfloat16 for computations, a setup that potentially accelerates training and inference, and curtails memory requirements. The method 600 continues to step 650,

[0180] In step 650, the low-rank model is trained using the base model and the custom training data. Once trainedbased on custom training data, the low-rank model is a custom low-rank model. The base model, which may be an application-specific fine-tuned model, for exmaple, may have a configuration as specified above, is then loaded along with an initialized low-rank adapter or other PEFT model (“low-rank model”). The low-rank model (e.g., the low-rank adapter) is then trained along with using output generated by the pretrained application- specific model using a transformer-based trainer, such as the HuggingFace transformers’ trainer (often specified as transformers.Trainer).

[0181] Low-rank adapter training integrates compact, task-specific adapter layers into pre-existing large-scalelanguage models and employs a low-rank factorization of weight matrices to significantly reduce the parameter space. This results in enhancing computational efficiency and storage requirements while maintaining the model’s versatility across diverse tasks. More specifically, a low-rank adapter the power of pre-existing large- scale language models. These adapters facilitate the adaptation of the model to specific tasks without necessitating changes to the original weights. The low-rank adapter method particularly employs low-rank approximation or factorization of the weight matrices found in these adapter layers. This factorization approach creates two smaller matrices that, upon multiplication, approximate the original weight matrix, thereby significantly cutting down the number of parameters and enhancing efficiency. The training process includes keeping the original model weights static (unchanged) and only training the adapter layer weights.

[0182] In one example, the low-rank adapter is initialized through use of a LoraConfig library (a configuration classthat stores low-rank adapter configurations) using the PEFT library, such as shown below:

[0183] In the LoraConfig, the argument “r” is used to specific a low-rank adapter attention dimension (type: int), theargument “target_modules” is used to specific a low-rank adapter attention dimension (type: Union[List[str],str]), the “lora_alpha” argument is an integer that sets the alpha parameter for low-rank adapter scaling, determining the scale of the low-rank adapter transformation. lora_dropout is a float that specifies the dropout probability for low-rank adapter layers, applying a regularization technique to prevent overfitting. fan_in_fan_out is a boolean, which should be set to True if the layer to replace stores weight like (fan_in, fan_out). bias is a string specifying the bias type for Lora and can be "none", "all" or "lora_only". modules_to_save is a list of strings, denoting modules apart from LoRA layers to be set as trainable and saved in the final checkpoint. layers_to_transform can be a list of integers or an integer, indicating the layer indexes to transform. If this argument is specified, the LoRA transformations will be applied on the layer indexes mentioned in this list. If a single integer is passed, the LoRA transformations will be applied on the layer at this index. layers_pattern is a string representing the layer pattern name, used only if layers_to_transform is different from None and if the layer pattern is not in the common layers pattern.

[0184] According to one embodiment, training arguments are provided that may be used by the trainer for training thelow-rank model.

[0185] Using the training configuration above with the HuggingFace™ trainer, the following python code then executesthe training process:

[0186] The trained model may then be saved in the custom low-rank model data store 446. The method 600 thenends.

[0187] With reference to FIG. 9, there is shown an embodiment of a method 700 of training a plurality of low-rankmodels, each of which is for use with a pretrained LLM, such as an application-specific model.

[0188] The method 700 begins with step 710, wherein custom user data is received from one or more custom dataparties. This step is analogous to step 610 of the method 600 and the discussion of the step 610 is hereby incorporated and attributed to the step 710 to the extent that discussion is not inconsistent with the discussion of the step 710. The custom user data received in this step is to be used for training a plurality of low-rank models. In some embodiments, the custom user data is received from a plurality of different custom data parties; however, in at least some embodiments, the custom user data is received from a single custom data party, such as the custom data party 404a. The method 700 continues to step 720.

[0189] In step 720, custom training data is generated for each of the plurality of low-rank models to be trained basedon the custom user data. In embodiments, such as where there are three low-rank models being trained, first custom training data (also referred to as a “first custom training dataset) is generated based on the custom user data for a first low-rank model; second custom training data is generated based on the custom user data for a second low-rank model; and third custom training data is generated based on the custom user data for a third low-rank model. The step 620 of the method 600 discussed generating custom training data based on custom user data and that discussion of the step 620 is hereby incorporated and attributed to the step 720 to the extent that discussion is not inconsistent with the discussion of the step 720. The method 700 then continues to step 730.

[0190] In step 730, common inputs amongst the custom training data for the plurality of low-rank models are identified.For example, referring to the example above of PCTTD-1, the custom training data may include hard-coded or predetermined questions or other static prompts, such as “What is the name of your business?”. For example, the first custom training data may include an entry: “Q: What is the name of your business?”, “A: John’s Lumber”; the first custom training data may include an entry: “Q: What is the name of your business?”, “A: Barbara’s Lemonade”. Here, the input (“Q: What is the name of your business?”) is the same and is considered a common input. According to embodiments, the custom training data is bifurcated into two groups, such as is discussed below.

[0191] Common input prompts shared across multiple custom training datasets are systematically identified, such asthrough hash-and-compare techniques or simply string comparison techniques. These shared prompts, despite being associated in each’s respective custom training dataset with varied outputs or responses, serve as a basis for segregating the data. Consequently, the custom training data is bifurcated into two distinct sets: a common-input training dataset and a varying-input training dataset. The common-input training dataset encompasses all instances bearing the shared prompts or inputs, regardless of the outputs or responses. Conversely, the varying-input training dataset includes the remaining instances characterized by unique prompts. This stratified approach to data organization paves the way for targeted model training and facilitates the extraction of insights from both shared and unique data interaction, as discussed below. The method 700 continues to step 740.

[0192] In step 740, the low-rank model is trained using the pretrained LLM and the custom training data. This stepmay use training configurations and operations similar to those used in the step 650 of the method 600, and that discussion of the step 650 is hereby incorporated and attributed to the step 740 to the extent that discussion is not inconsistent with the discussion of the step 740. Indeed, according to the present embodiment, the step 740 includes carrying out two training phases: a shared training phase 742 and an individualized training phase 744. In the shared training phase 742, the principle of frozen weights used for the pretrained LLM for low-rank adaptation is relied upon. More particularly, the shared training phase 742 utilizes the common-input training dataset. Again, here, the pretrained LLM is frozen, ensuring that a common input consistently produces the same output. This allows for a single forward pass through the LLM for each shared prompt, generating an output that serves as a consistent input for each of the plurality of low-rank models.

[0193] In at least one embodiment, low-rank adaptation is used whereby the following Equation (1) is modified intoEquation (2) in order for the shared training phase:

[0194] In general, the rank r is and represents the frozen base model parameters. Equation (2)represents the operation for each low-rank adapter i, with : this is the output for the i-th low-rank adapter, after the adapter has been trained on its custom data; : this is the original, unchanged weight matrix that is used by all the adapters; : this is the input for the i-th adapter, and may be a vector representation of a text input, an image, or some other kind of data, in embodiments; : this is the change in the weight matrix for the i-th adapter, which results from training the adapter on its custom data, and this matrix represents the specific, learned adaptations of the i-th adapter. Equation (2) shows how each low-rank adapter applies its own learned adaptations to the original, unchanged weight matrix , in order to produce its output h_i from its input . The process of training each adapter involves backpropagating the error of with respect to the target output, and adjusting in order to minimize this error. This learning process does not affect , which remains the same for all adapters.

[0195] If is always the same and so is , this may be taken advantage of to simplify the computations and savecomputational resources thereby improving the efficiency of the computerized training system. For example, pre-computing and denote it as a new variable, that may be called . Therefore, Equation (2) can be rewritten as:In this case, is a constant and does not change. The only part that varies for different adapters is . By precomputing , you avoid performing the matrix-vector multiplication multiple times, which could lead to significant computational savings, especially if you have many adapters.

[0196] After the shared training phase 742, an individualized training phase 744 proceeds, where the low-rank modelis trained using the varying-input training dataset. Each low-rank model is individually trained on its respective unique prompts during this phase as the prompts in the varying-input training dataset vary. The main advantage of executing the shared training phase 742 one lies in its computational efficiency. Despite the varied outputs for different training datasets, the pre-trained LLM, a significantly large and complex model, only necessitates a single forward pass for each common input prompt. This single-pass mechanism eliminates the need for multiple forward operations through the LLM, significantly reducing computational demands and enhancing the overall training efficiency, all while preserving the consistency of responses for common prompts across the multiple low-rank models.

[0197] It will be appreciated that, although the present embodiment discussed the shared training phase 742 as beingperformed first, in other embodiments, the individualized training phase 744 is performed before the shared training phase 742. The method 700 then ends.

[0198] The trained low-rank models that are obtained as a result of the method 700 may be stored in a suitable datastore, such as the low-rank model data store 466.

[0199] With reference to FIG. 10, there is shown an embodiment of a method 800 of training a plurality of low-rankmodels, each of which is for use with a pretrained LLM, such as an application-specific model.

[0200] The method 800 begins with step 810, wherein custom user data is received from one or more custom dataparties. This step is analogous to step 710 of the method 700 and the discussion of the step 710 is hereby incorporated and attributed to the step 810 to the extent that discussion is not inconsistent with the discussion of the step 810. In embodiments, tens, hundreds, or even thousands of different custom training requests are received, where each custom training request specifies a custom user data set that is custom user data for training a low-rank model. The custom training requests are received via the custom training API subsystem 434 from the custom data party 404a, the custom data party 404b, and the custom data party 404c. The custom user data is stored, such as at the custom user data store 436. The method 800 proceeds to step 820.

[0201] In step 820, custom training data is generated based on the custom user data. This step is analogous to step720 of the method 700 and the discussion of the step 720 is hereby incorporated and attributed to the step 820 to the extent that discussion is not inconsistent with the discussion of the step 820. The method 800 continues to step 830.

[0202] In step 830, the custom training data is pooled or grouped according to commonality amongst inputs. Customtraining data that has common inputs may be identified and grouped together so that these custom training data sets may be trained using the two-phase process described herein. In embodiments, the custom training data is stored in a pooled custom training data store where custom training data is pooled together based oncommonality of inputs and the pretrained LLM to be used for training (e.g., an application-specification fine- tuned LLM). The method 800 continues to step 840.

[0203] In step 840, a plurality of low-rank models that are trained using the custom training data. In embodiments, theplurality of low-rank models are those that are associated with custom training data that is commonly pooled. The training performed in this step 840 is analogous to step 740 of the method 700 and the discussion of the step 740 is hereby incorporated and attributed to the step 840 to the extent that discussion is not inconsistent with the discussion of the step 840. In embodiments, the shared training phase 742 is used to train the plurality of low-rank models, and the individualized training phase 744 is performed without regard to the pooling. The method 800 ends.

[0204] Thus, according to embodiments, the pretrained LLM forms the shared backbone, serving multiple low-rankmodels trained for various tasks or datasets. Every adapter possesses the ability to fine-tune the shared representations from the LLM to its distinct task or use case (according to the training dataset), allowing for specialized adaptations. An advantage of this approach lies in its computational efficiency. The pretrained LLM, due to its frozen state during training, processes inputs just once, and the output is repurposed across multiple tasks or datasets by linking to distinct adapters. This streamlined operation circumvents the need for multiple forward operations through the computationally intensive LLM. As a result, computational demands are substantially reduced, contributing to enhanced training efficiency. Moreover, this method maintains the consistency of responses to common prompts across the multitude of low-rank models, ensuring uniformity in the shared knowledge space.

[0205] With reference to FIG. 11, there is shown an embodiment of a method 900 of training a plurality of low-rankmodels, each of which is for use with a pretrained LLM, such as an application-specific model.

[0206] The method 900 begins with step 910, wherein training data sets are obtained. The training data sets may bethe custom training data sets generated above, or may be other training data sets. The method 900 continues to step 920, where custom training data is pooled or grouped according to commonality amongst inputs. This step 920 is analogous to step 830 of the method 800 and the discussion of the step 830 is hereby incorporated and attributed to the step 920 to the extent that discussion is not inconsistent with the discussion of the step 920. The method 900 continues to step 930, where a plurality of low-rank models that are trained using the custom training data. This step 930 is analogous to step 840 of the method 800 and the discussion of the step 840 is hereby incorporated and attributed to the step 930 to the extent that discussion is not inconsistent with the discussion of the step 930.

[0207] According to embodiments, there is provided a speech-adapted LLM message communication system havinga speech-adapted LLM message generation system, according to one embodiment. The LLM message generation system is analogous to the LLM message generation system 412, except that the speech-adapted LLM message generation system is expressly described as being adapted to receive user audio as input, which is then converted and used to generate a response message based on a forward pass of the message generation network. Besides for the below discussed differences, the speech-adapted LLM message generation system is analogous to the LLM message generation system 412 and that discussion of the system412 is incorporated and attributed to the speech-adapted LLM message generation system to the extent that discussion is not inconsistent with the express discussion below of the LLM message generation system. The speech-adapted LLM message communication system is analogous to the LLM message communication system 410, and that discussion of those like components is hereby incorporated and attributed to the speech- adapted LLM message communication system.

[0208] The LLM message generation system includes a speech-based message generation network that uses audioembeddings to generate a response message, and such relevant discussion of those components above is hereby incorporated and attributed to these like components. As used herein, “audio embeddings” or “audio feature embeddings” are embeddings that are generated for an audio feature. As used herein, an “audio feature” is a discernible characteristic or element of an audio signal, which may be a “non-speech audio feature” or a “speech audio feature”. A “non-speech audio feature” is an element associated with general sound properties and musical elements that can be extracted and represented in a numerical format, and this may include, but is not limited to, frequency components, temporal dynamics, spectral content, acoustic intensity, harmonic content, and tonality. These features are particularly applicable to tasks like sound classification, music information retrieval, and noise reduction. On the other hand, a “speech audio feature” refers to discernible characteristics within vocal communication, including aspects such as speech volume, accent characteristics, speaker identity, tonal inflections, rhythm, and rate of speech. These features are crucial for applications such as speaker recognition, speech recognition, and accent identification. The term “audio embeddings” refers to embeddings that are generated for non-speech and / or speech audio features. These embeddings encapsulate and encode these diverse features in a lower-dimensional space, which enhances their computational manageability and subsequent application in machine learning or deep learning algorithms.

[0209] In embodiments, the speech-to-text adapter (employed as a part of the input builder 116, for example) employsautomatic speech recognition (ASR), such as NVIDIA NeMo (see Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and others. Conformer: convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100, 2020).

[0210] In embodiments, a speech diarization technique is used in order to provide a speaker label for audio data sothat speech audio features and / or non-speech audio features are assigned to a particular user. Speech diarization may also be used when preparing custom user data into training data for training a low-rank model or head model. In such a case, speaker diarization is used to identify which portions of the audio were uttered by the user (or another particular user). This is useful for identifying pertinent data to use for training and may be tailored based on the specific training to be performed.

[0211] For example, the NeMo speaker diarization system may be used to perform the speaker diarization technique.The NeMO speaker diarization system is constructed of several modules, including a Voice Activity Detector (VAD), Speaker Embedding Extractor, Clustering Module, and a Neural Diarizer. The VAD is a trainable model that identifies the existence or non-existence of speech, generating temporal markers for the duration of speech activity within the provided audio recording. The Speaker Embedding Extractor is also a trainable model andis responsible for deriving speaker embedding vectors that encapsulate the unique vocal characteristics from the raw audio input. Further, the Clustering Module is a non-trainable element, which categorizes speaker embedding vectors into distinct clusters. Lastly, the Neural Diarizer, another trainable model, estimates speaker labels based on the provided features. The evaluation of speaker diarization may be executed in two distinct modes contingent upon the VAD settings: the oracle VAD mode, which involves speaker diarization based on verifiable VAD timestamps, and the system VAD mode, where speaker diarization is contingent on the outputs from a VAD model, at it will be appreciated that the model may be selected according to implementation details and the application in which it is to be used.

[0212] The LLM message generation system includes the input builder 416, but with a speech to text adapter that isused for processing an audio message from a user (or “user audio message”) and generating an inference model input, which may include the transcribed user message, extracted audio features, and processed conversation history data. The inference model input is passed into both the low-rank model 422 and the base model 420 in order to generate respective outputs that are then combined.

[0213] The dual-selectable model framework is a framework for a LLM message generation system that enables theLLM message generation system to select a first model and a second model for use in generating messages. For example, in the present embodiment, the dual-selectable model framework is implemented by the LLM message generation system, thereby enabling the LLM message generation system to generate an output (response message) based on using a LLM and two selected models, the first one being a low-rank model (the selected low-rank model 422 in FIG.12) and the second one being a head model (the selected head model).

[0214] In embodiments, the message generation network of the LLM message generation system implements a dual-selectable ensemble network, such as the dual-selectable model framework 413'',413''', and may include a fine-tuned head selector that selects a fine-tuned head from a set of fine-tuned heads for use as a part of the message generation network. The fine-tuned head may be used for adjusting the output indicative of a message based on, for example, a predefined tone such that this adjusted output embodies the predefined tone in expressing the message.

[0215] In embodiments, a me model is employed by the message generation network by selecting a low-rank modelthat was trained based on conversation history for the user in response to user input, such as a user login or a user selection (e.g., a user selecting to use the “MeChat” option). The selected low-rank model 422 (or me model) in this example is then used along with the base model 420 to generate an initial output, which is then used by the selected fine-tuned head (e.g., a tone conditioning head) to produce the network output (output of the message generation network), which may then be provided to the user. For example, the user device 414 includes a computer app, such as a browser or mobile application (e.g., iOS app, Android app), that provides a graphical user interface (GUI), including a conversation window showing conversation history for a conversation session. In one embodiment, the GUI provides an option for a user to talk with themselves, through use of a MeChat in which the user’s me model is used to generate messages that are then provided to the user device 414.

[0216] With reference again to FIGS.12 and 13, in embodiments, each LLM message generation system 412'',412'''is configured to use the dual-selectable model framework 413'', 413''' in order to select the me model (or other low-rank model) for the user, and is further configured to select a fine-tuned head, which may be based on user input or automatically selected by the system. With reference to FIG.15, there is shown an automated text conversation graphical user interface (GUI) 1801 used as a part of a client application for facilitating a text conversation with a user, such as for display on the user device 14,114,414. The text conversation GUI 1801 includes an automated text conversation screen 1802 in which automatic messages are generated for a conversation 1804. In particular, the conversation 1804 is represented by a set of automatically-generated messages 1806 and a set of user messages 1808. The GUI 1801 further includes a user message input 1809, which may be a text input, such as a text field or text area that a user may type characters into. The set of automatically-generated messages 1806 includes three exemplary automatically-generated messages 1812a– c generated by a message generation network and an initial message 1811 which may or may not be generated using the message generation network. The set of user messages 1808 includes four exemplary user messages 1814a–d. These messages form the conversation 1804 between the user and an automated message generation network, such as through use of the LLM message generation system 412'' of FIG.12 or the LLM message generation system 412''' of FIG.13. The automated text conversation screen 1802 of the exemplary embodiment also includes a tone control graphical controller 1821 that includes a plurality of selectable options 1822a–e, particularly, in the present embodiment, for selecting a tone that should be used for generating the automatically-generated messages for the conversation 1804. In FIG.15, the second option 1822b corresponding to the tone “Excited” is presently selected and so that the automatically-generated message that is to be generated in response to the user message 1814d uses an excited tone. In at least some embodiments, data is sent to the LLM message generation system from the user device in order to indicate which tone is selected, and this data may be referred to as tone selection data or, more generally, user-configurable options. According to embodiments, the LLM message generation system 412'',412''' uses the fine-tuned head selector 420'' to select a fine-tuned head (e.g., one of the fine-tuned head networks 226) based on user-configurable options, such as a selected tone for the response message.

[0217] In one embodiment, the message generation network supporting the GUI 1801 may be the LLM messagegeneration system 412'',412''', which uses the dual-selectable model framework 413'', enabling the user messages to be tailored to the user (through use of a custom low-rank model, such as the me model) while also modifying the tone output by that custom model, thereby providing a toned custom user generation network 419'',419'''. In embodiments, this enables a user to emulate a conversation with themselves while being able to modify the tone of the conversation at runtime through selection of the tone using the tone control graphical controller 1821 of the GUI 1801. For example, a real individual, such as the user or a pop culture star, or a fictional character, such as a character from a movie or television series, may be the target of the training and the training data may be selected accordingly. For example, in embodiments, a user may upload conversation history data or other data that includes content, such as text or audio content, generated by the user in order to have a model trained based thereon—this data is referred to as user-submitted training data. The user-submitted training data may be data pertaining to content generated by a particular individual, such as a real individual (e.g., the user, the user’s grandparent, Kobe Bryant), or a fictional character. The user-submitted training data is used to train a low-rank model and then a plurality of tone conditioning models may be used as well, such as through selection of a corresponding tone-condition fine-tuned head. This enables, for example, two dimensions of customization, each exploiting benefits of fine-tuning rather than simply using prompt engineering and few-shot learning. A user may use the GUI 1801 to begin a start session with Kobe in a Happy tone, or may select a different tone, such as Excited, during runtime. The fine-tuned head is the swapped by the fine-tuned head selector 420''' so that a new head corresponding to Excited is used as the active fine-tuned head 421'''.

[0218] According to embodiments, any of the LLM message generation system 12, 100, 200, 201’, 412, 412', 412'',412''' may be used for the GUI 1801.

[0219] With reference to FIG.14, there is shown an embodiment of a method 1100 of generating a response messageusing a large language model (LLM), such as an application-specific model. The method 1100 is carried out any one of the above-discussed LLM message generation systems, such as the LLM message generation system 12, 100, 200, 201’, 412, 412', 412'', 412'''.

[0220] The method 1100 begins with step 1110, wherein an audio message for a conversation session is obtained.In embodiments, this step involves obtaining audio data captured by a digital audio recording device, such as a microphone, embedded or connected to a user’s digital device, which could be a smartphone, computer, tablet, smart speaker, or any other capable electronic gadget. The conversation session may be initiated when the user speaks into the microphone, thereby generating an audio message.

[0221] In embodiments, this user audio message is captured and digitally encoded into a format suitable for furtherprocessing, and the encoding may be performed using various standard or proprietary audio coding formats such as Pulse Code Modulation (PCM), Advanced Audio Coding (AAC), or others. It should be noted that the user’s audio message may be an open-ended conversation, a specific question, a command, or any other type of verbal communication that can be initiated by the user.

[0222] The obtained audio message is stored temporarily in a buffer or similar type of memory storage, ensuring it’savailable for subsequent steps in the method. The audio message may also be timestamped and associated with the particular conversation session for proper context and response generation. This step, therefore, provides the necessary input data that drives the remainder of the patented method. The method 1100 continues to step 1120.

[0223] In step 1120, conversation history data for the current conversation session is obtained. This step facilitatesthe contextual understanding necessary for generating an appropriate and meaningful response. The conversation history data typically comprises previous exchanges or interactions between the user and the system within the specific conversation session. This data may include, but is not limited to, prior audio messages from the user, text-based messages from either party, system-generated responses, and associated metadata such as time and date of the conversation elements, or other relevant information. To retrieve this conversation history data, the system accesses a dedicated storage medium, which could be a local database,cloud-based storage, or other appropriate data repositories. The storage medium would have been continually updated during the conversation session, recording each interaction to form a coherent and sequential dialogue history. The acquired conversation history data plays a pivotal role in enabling the system to maintain continuity and relevance in its responses, by keeping track of the ongoing dialogue and the user’s intent. Furthermore, the historical data is structured and prepared for subsequent processing steps, ensuring it can effectively inform the generation of the response message. The method 1100 continues to step 1130.

[0224] In step 1130, an inference model input is generated, which is based on the combination of the user audiomessage and the obtained conversation history data. This step is pivotal as it transforms the raw input data into a format that is suitable for the Language Learning Model (LLM) to process.

[0225] The user audio message may first be converted from its encoded format into a textual representation; inembodiments, this transformation is achieved using an Automatic Speech Recognition (ASR) system, which transcribes the spoken words into written text, thus allowing the LLM to process the message.

[0226] Furthermore, the user audio message undergoes a process of feature extraction to isolate the relevant “speechaudio features” and “non-speech audio features”. This process includes identifying aspects such as the tone of the message, represented by various tone classes (e.g., angry, happy) and any other pertinent audio characteristics. Once these features are extracted, they are transformed into audio embeddings, numerical representations of the specific attributes of the audio message. Each embedding represents a different characteristic of the audio message; one might symbolize the spoken words, while another might represent the tone. The generated audio embeddings are then tokenized to create an inference model input that the LLM can effectively process. During this tokenization step, each word or phrase from the text transcription is assigned a corresponding token that the LLM recognizes. Simultaneously, the tone class, as identified from the audio features, is mapped to its corresponding token that was predefined during the training phase. The resultant tokenized input is a combination of both word and tone tokens, reflecting not just the semantic content of the user audio message, but also the emotional context. This tokenized input, alongside the obtained conversation history data, forms the basis for generating the inference model input in step 1130. Thus, the system is enabled to generate a response that is both contextually apt and emotionally sensitive, leading to a more comprehensive and nuanced user interaction.

[0227] In embodiments, simultaneously, the conversation history data, which comprises textual data and metadatafrom the ongoing conversation session, may be prepared for incorporation into the inference model input. The preparation may involve steps such as tokenization, where the text and / or audio is broken down into smaller pieces (tokens), and other pre-processing tasks such as removing unnecessary elements, standardizing the format, and more. In embodiments, the tokenized history for the conversation session is stored in memory so that it may be accessed during future message generation processing for the conversation session. The conversation history embeddings refer to embeddings representing one or more inference inputs generated based on a user message and for use in a message generation network to generate a response message. And, conversation history tokens refer to tokens representing one or more inference inputs generated based on a user message and for use in a message generation network to generate a response message.

[0228] Following the preparation, the transcribed user message, the extracted audio features, and the processedconversation history data are combined to create the inference model input. In embodiments, transcribed user message embeddings, extracted audio feature embeddings, and conversation history embeddings are concatenated to generate the inference model input; however, other techniques may be used according to embodiments, such as other relevant data fusion techniques. The specific method used can depend on the architecture of the LLM and the format it requires for inputs.

[0229] The final inference model input, therefore, contains both the current query from the user and the context,including audio feature context as represented by the extracted audio tokens, necessary for the LLM to generate an appropriate response. This input is subsequently fed to the LLM in the next step of the process. The method 1100 continues to step 1140.

[0230] In step 1140, a response message is generated based on an inference output. This output is derived from theinputting of the inference model input into two distinct models: a low-rank model and a base model. The utilization of these two models enhances the efficiency and accuracy of the response generation, enabling the presently-discussed method and pipeline to provide a live or real-time LLM message generation system, such as a live chatbot responds to the user with low latency. As used herein, a “live” LLM message generation system is a LLM message generation system that, on average, is usable to generate a response message for a user message within thirty (30) seconds from the time the user message is received. As used herein, a “real- time” LLM message generation system is a LLM message generation system that, on average, is usable to generate a response message for a user message within ten (10) seconds from the time the user message is received. For example, if for a first system, 100 samples are taken, with the average being 30 seconds or less, then the first system is live; and, for a second system, 100 samples are taken, with the average being 10 seconds or less, then the second system is real-time. Any one or more of the LLM message generation systems discussed herein may be implemented as a live LLM message generation system, according to embodiments.

[0231] The inference model input, which may incorporates the user’s current query and the preceding conversationhistory, is first fed into the low-rank model. Generally, in embodiments, the low-rank model is designed to capture the salient features and patterns in relevant or pertinent data and quickly generate an initial inference output. The low-rank model can be a simplified or distilled version of a larger model, allowing for a balance between computational efficiency and response accuracy.

[0232] Also, the same inference model input is fed into the base model. Generally, in embodiments, the base modelis a more comprehensive and detailed model, which can capture complex patterns and relationships in the data, leading to a more refined and precise inference output. This model may require more computational resources compared to the low-rank model, but it ensures the highest quality of response generation.

[0233] The outputs from the low-rank model and the base model may then be aggregated. This aggregation can beachieved through various techniques such as averaging, weighted averaging, or other appropriate methods, depending on the specific architecture and requirements of the system. The aggregated output thus captures both the speed and efficiency of the low-rank model and the depth and precision of the base model.

[0234] Finally, the aggregated output is transformed into a response message. This transformation could involveconverting the output into natural language text, which can then be synthesized into speech, displayed as text, or delivered in another suitable format. In embodiments, this transformation involves determining natural language (NL) text and one or more audio feature tokens from the output, and this combined data (the determined NL text and the one or more audio feature tokens) is referred to as audio feature-enhanced NL text. The audio feature-enhanced NL text is then converted into audio data that represents spoken NL audio for speaking the NL text while incorporating the determined one or more audio feature tokes. For example, if the audio feature token identified a tone of anger in the user's message, the system could incorporate a matching tone into the audio data for the response message. Thus, when the audio feature-enhanced NL text is converted into audio data, the resultant spoken message may carry a tone of empathy or urgency, appropriately mirroring or responding to the user’s original emotional tone. This ensures that the system’s responses are not only contextually accurate, but also emotionally attuned, leading to a more natural and effective user interaction. The response message, therefore, may be a direct and contextually-aware response to the user’s initial audio message, based on both the immediate query and the preceding conversation history. The method 1100 ends.

[0235] In embodiments, the LLM message communication system 410 and the LLM message generation system 412uses a sequence-to-sequence transformer model that is trained on many different speech processing tasks, including multilingual speech recognition, speech translation, spoken language identification, and non-speech audio features. For example, a present embodiment employs OpenAI’s Whisper technology (A Radford, JW Kim, T Xu, G Brockman, C McLeavey, I Sutskever, Robust speech recognition via large-scale weak supervision, arXiv preprint arXiv:2212.04356), but modified to incorporate the non-speech audio features, which are often not addressed and simply ignored by current implementations. In embodiments, all of these tasks are jointly represented as a sequence of tokens to be predicted by the decoder, allowing for a single model to replace many different stages of a traditional speech processing pipeline and further incorporating information held within non-speech audio feature tokens. In embodiments, in order to generate audio feature tokens, non-speech audio features are extracted from audio through use of a convolutional neural network (CNN) that is carried out over the audio data, such as through using a raw waveform of the audio to extract latent audio non-speech audio feature representations.

[0236] The sequence-to-sequence transformer model includes adapting the multitask training to use non-speechaudio feature tokens. According to embodiments, the audio feature-based transformer disclosed herein seeks to marry the efficiencies of transformer-based LLMs, while incorporating rich information derived using speech emotion recognition (SER) techniques. In embodiments, there is provided a process of generating a non- speech audio feature token in which an SER technique is used to identify an emotion conveyed in audio, such as in a waveform representing sound uttered by a user, and then the emotion is mapped to a predefined non- speech audio feature token, which is used as a part of a tokenized input that is passed into a transformer LLM. Thus, according to embodiments, this enables use of various transformer LLMs without having to further refineand customize transformer nodes of the transformer LLM while still incorporating rich input derived from the audio.

[0237] In embodiments, non-speech audio feature tokens are a part of a predefined token set, such as a plurality ofpredefined tokens that the base model (e.g., application-specific base model) is configured to use. According to embodiments, a transformer-based large language model is trained using a predefined token set that includes at least one non-speech audio feature token using a process that begins with the selection and definition of the desired tokens. Generally, for text-based transformers, these tokens can represent any sequence of text the user wishes the model to treat as a single, indivisible unit. It can be a unique word, a phrase, or a complex syntactic construct that is relevant to the specific application for which the language model is being trained.

[0238] Audio signals carry a wealth of information that can provide context beyond just the words being spoken. Someexamples of audio features that can provide emotional or contextual cues, without being directly related to language or speech, include: (1) speaker tone and pitch: changes in the speaker’s tone or pitch is often indicative of one or more of a variety of emotions. For example, a raised pitch might indicate excitement or anger, while a lower pitch might suggest sadness or tiredness; (2) volume: the loudness of a speaker’s voice can also provide emotional cues, such as a louder volume might suggest anger or enthusiasm, while a quieter volume could indicate sadness or fear; (3) speech rate: the speed at which a person speaks can also indicate their emotional state, such as rapid speech might suggest excitement or nervousness, while slower speech could suggest calmness or possibly depression; pause duration: length and frequency of pauses in speech can provide clues about a speaker’s emotional state or the context of the speech, such as longer or more frequent pauses could indicate uncertainty or hesitation; (5) voice quality: features like breathiness, hoarseness, or nasality in a speaker's voice can provide information about their emotional state or physical condition; (6) non- verbal sounds: sounds that are not words, like laughter, sighs, cries, or other vocalizations that are classifiable to a particular audible expression also oftentimes provide important cues about a speaker’s emotional state; and (7) background noise: sounds in the background of an audio signal can provide contextual information, such as, for example, the sound of traffic might indicate that the speaker is outdoors, while the sound of a computer keyboard might suggest an office environment. These features can be analyzed using various audio processing techniques, such as spectral analysis, to provide valuable information about the speaker's emotional state or the context in which the speech is taking place. Such features are often used in fields like emotion recognition, health monitoring, and human-computer interaction. Of course, according to various embodiments, a variety of different non-speech audio feature tokens may be used as part of the predefined set of tokens.

[0239] Following the selection of custom tokens, the next step involves their introduction into the training dataset. Thisprocess could require substituting occurrences of the selected sequences of text in the existing dataset with the custom tokens. Alternatively, it could also involve supplementing the dataset with additional examples that incorporate the custom tokens. The goal here is to provide sufficient examples of each custom token within awide range of contexts. This diversity in exposure allows the model to learn the meaning and usage of these tokens effectively.

[0240] To prepare the data for the training of the language model, the tokenization algorithm needs to be adjusted torecognize these custom tokens. This adjustment could require modifying the algorithm to ensure each custom token is treated as an unbreakable unit. Another approach could be to furnish the algorithm with a list of custom tokens to augment its existing vocabulary. Ensuring that the tokenization process recognizes and respects these custom tokens guarantees they are not broken down into smaller, less meaningful units during the data preparation phase.

[0241] Training the language model follows the data preparation phase. This process involves feeding the tokenizeddata into the model and iteratively adjusting the model's parameters. The aim here is to reduce the difference between the model's predictions and the actual outcomes in the dataset. Depending on the specific needs and resources available, the model may be trained from scratch, or an existing model may be fine-tuned using the prepared dataset. The latter approach, fine-tuning an existing model, is often advantageous as it leverages the knowledge the model has already gleaned from previous training, while adapting it to recognize and effectively utilize the custom tokens.

[0242] Once the model has been trained, it’s important to assess its performance and fine-tune it as necessary. Themetrics for this evaluation will depend on the specific application, but they could include measures of the model's accuracy in predicting or generating text that includes the custom tokens. In essence, this technique provides an effective way to customize a large language model's vocabulary to encompass specific tokens of interest, thereby significantly enhancing its functionality for various applications. By ensuring that certain sequences of text are treated as indivisible units, the model is better equipped to understand and generate these sequences accurately and meaningfully.

[0243] With reference to FIG. 16, there is shown a method 1600 of generating a response message for a usermessage, according to one embodiment. The method 1600 begins with step 1610, wherein a user message is received. The user message is received from the user device 14. The user device 14 is illustrated as a smartphone, but the user device 14 may be any of a variety of internet of things (IoT) devices, such as those with user-installable applications and / or internet browsing capabilities, including personal computers (e.g., desktops, laptops), smartphones, tablets, automotive infotainment units, etc. The user message 1610 may be received as an audio message or as a text or written message, such as one received via conversation with a chatbot. For example, a web application for a browser may be used for providing a GUI for facilitating a conversation between the user and the LLM message generation system 12. The method 1600 continues to step 1620.

[0244] In step 1620, a vector data store is queried for a content identifier based on the user message. As discussedabove, the vector data store 108 includes textual gloss data for content received by the system 12. In one embodiment, the vector data store 108 is queried through performing a nearest neighbor search and, in some embodiments, may employ a nearvector (near-vector) and / or neartext (or near-text) operation. For example, the textual gloss data may include a plurality of textual gloss data entries for a given portion of content, suchas a given paragraph. A textual gloss data entry includes textual gloss data that includes text representing or characterizing in some manner a portion of content (a “textual gloss”). The textual gloss data entry further includes, at least in the illustrated embodiment, a content identifier, which identifies the portion of content, such as a section and / or paragraph number, to which the textual gloss relates. The textual glosses may include phrases, sentences, or even larger bodies of text, or may only include a single term, for example. The nature of the textual glosses is dependent upon the content and particular application in which the present technology is used.

[0245] In some embodiments, the textual glosses include one or more glossary terms associated with a contentidentifier, and these glossary terms may be concepts that relate to the content identified by the content identifier. Furthermore, in embodiments, the glossary terms are generated according to a custom prompt that is specified by a user through use of the content retrieval API subsystem 106 or other similar API, for example. In one embodiment, a neartext search is run using Weaviate’s built-in neartext querying, where a query is run using the user message as input against textual gloss data in the vector data store 108. The most relevant entries are returned and the results include content identifiers. The method 1600 continues to step 1630.

[0246] In step 1630, a portion of the content is obtained based on the content identifier. The content identifier obtainedin the previous step is used to query a data store, such as the content data store 110, for the portion of the content associated with the content identifier, which may correspond to one or more paragraphs, pages, sections, chapters, spreadsheets, etc. of the content. In one embodiment, the content is stored in the content data store 110 as a part of a separate vector data store. The vector database may be queried using a WHERE operation based on the content identifiers. And, in embodiments, the user message may be used to query the content stored in the vector database so that the full content, as opposed to the textual gloss data only, is used for searching for relevant information within the content. This obtained content, which collectively may be referred to as a portion of the content, is then used as input, as discussed below. The method 1600 continues to step 1640.

[0247] In step 1640, a language model input is generated based on the user message and the portion of the content.The language model input is used for inference in order to generate a response message. In at least one embodiment, the language model input includes the user message and the portion of content obtained in step 1630. In embodiments, such as where an automated conversation between the user and the system 12 is performed, the language model input further includes message history data for the current conversation. The language model input is represented as a text string that is to be input into a LLM, such as into OpenAI’s ChatCompletion API (which may be used to implement the message generation network 118 in embodiments). The method 1600 continues to step 1650.

[0248] In step 1650, a response message is generated based on inputting the language model input into a languagemodel whereby the language model performs inference using the language model input. As mentioned above, the language model input is input into the message generation network 118 in order to generate a response message, which is then provided to the user. The method 1600 ends.

[0249] With reference to FIG. 17, there is shown an LLM message generation system 1700, according to oneembodiment, particularly wherein a technical document message generation computer application 1702 is provided whereby a user may provide natural language requests using the user device 14 and receive responses thereto. The technical document message generation computer application 1702 is implemented as a web application having a frontend 1704 provided a graphical user interface (GUI) 1706 and a backend 1708 providing message generation and other processing for the frontend 1704. The frontend 1704 is implemented using the React framework, which is built on JavaScript (JS), in the present embodiment; and, in the present embodiment, the backend 1708 is implemented using Flask™ and built on Python 3. Of course, in other embodiments, the frontend 1704 and the backend 1708 may be implemented using a variety of different libraries and platforms.

[0250] The backend 1708 implements and includes a message generation API 1710, which provides a networkinterface for the frontend 1704 to use when communicating with the backend 1708. The backend 308 further includes a message generation engine 1712, which may correspond to a LLM message generation system, such as the LLM message generation system 12 discussed above. Thus, the discussion regarding the LLM message generation system 12 is hereby incorporated and attributed to the LLM message generation engine 1712 to the extent that discussion of the LLM message generation system 12 is not inconsistent with the discussion of the LLM message generation system 1712. The backend 1718 also includes a data store interface 1714, which is used to perform data store operations, such as, for example, storing data, deleting data, modifying data, or recalling data.

[0251] The message generation API 1710 provides an interface for network communications. The messagegeneration API 1710 is implemented using the Flask™ framework, which is a lightweight web application framework written in Python, and the API 1710 is built on Python 3, with a Flask™ server being defined in the Python file ‘service.py’ and specifying multiple endpoint modules 1716, specifically a chat end point, a document endpoint, a user endpoint, and a payment endpoint. Each of the endpoint modules is specified in a different Python file and includes one or more endpoint functions, each of which is a Python function mapped to an endpoint path. The endpoint path may be specified using a variety of methods or manners, such as using annotations or decoration (e.g., @chat_endpoint.route(" / chat_message", methods=["POST"]). Of course, this is but one example.

[0252] The message generation engine 1712 is used to provide services or processing according to the requestsreceived at the message generation interface 1710. The message generation engine 1712 is shown as including a variety of modules each corresponding to a separate Python file, including a ‘patent_util.py’ file for performing domain-specific utility functions and other domain-specific functionality, particularly, patent document and data processing in the illustrated embodiment. The message generation engine 1712 further includes a message generator Python file (‘message_generator.py’) that is used for implementing a variety of functionality for preparing input for a message generation network, causing the message generation network to be executed with the prepared input, and receiving and processing the output of the message generation network. In the present embodiment, this message generator module is used for generating and providingmessages for a chatbot provided by the application 1702. The message generation engine 1712 further includes a ‘summarizer.py’ Python file that is used for providing domain-specific message generation processes, such as for executing custom summarization of technical documents, such as patent specifications. For example, a summary of the text document may be included in the

[0253] The data store interface 1714 is used for providing a coherent and unified API for performing data storeoperations across different types of databases through specialized utility files. The ddb_util.py file contains a suite of functions and classes specifically tailored for interfacing with Amazon DynamoDB, a NoSQL database service. This utility enables operations such as storing, deleting, modifying, and recalling data within the DynamoDB environment, ensuring that data manipulation is optimized for the key-value and document database paradigms that DynamoDB supports. In parallel, the vdb_util.py file is dedicated to interfacing with Weaviate™, a vector database that facilitates machine learning-powered data storage and retrieval. This utility file encapsulates the unique functionalities required to interact with Weaviate’s vectorized data, allowing for operations that include the insertion of vectorized information, deletion based on vector or semantic criteria, updates to existing vector records, and the querying of data through vector search and semantic understanding. Together, these two utility files within the data store interface 1714 abstract the complexities of the underlying database technologies, providing a streamlined and consistent API for the end-user to perform necessary data operations efficiently, regardless of the database type in use. This design allows for the flexibility to switch between or simultaneously leverage the strengths of both traditional NoSQL data storage with DynamoDB and AI-enhanced vector data management with Weaviate.

[0254] With reference now to FIGS. 18-21, there are shown flowcharts depicting various processing that may beperformed by a LLM message generation system or engine, such as the LLM message generation system 12, 412, 1712, etc.

[0255] With reference to FIG. 18, there is shown a flowchart of a method 1800 of generating a response message inresponse to a user message, such as for purposes of implementing a chatbot, for example. In particular, the method 1800 is described in the present embodiment as a method for generating a response message based on inference of an LLM performed on an augmentative input comprised of the user message and augmentative data. In the illustrated embodiment described herein, the method 1800 is used as a part of implementing a technical document chatbot, such as a chatbot for communicating with a natural language assistant (through text messages, for example) that has access to contents or information concerning the technical documents. The technical documents may be patents, pre-grant patent application publications, government guidelines, state laws, government regulations, statutes, case law, other public documents, and / or uploaded documents from a user, which may be a variety of different material or content.

[0256] The method 1800 begins when a user message is received at 1810. This message is analogous to the step 1610of the method 1600 and that discussion is hereby incorporated and attributed to the step 1810 to the extent that discussion is not inconsistent with the discussion of the step 1810. After the user message is received, the method 1800 proceeds to step 1820.

[0257] In step 1820, a context (referred to in the present embodiment as a “chat context”) is determined. The contextmay be determined based on the user message and / or user input, such as where the user specifies one or more document or resource identifiers (collectively, also referred to as “content identifiers”), such as a patent number, attorney docket number for a patent application, DOI (digital object identifier) for a white paper or other publication, or assigned resource identifier. In at least one embodiment, a ‘context_manager.py’ Python file is used to implement Python functions that use regular expressions (referred to also as regexes or regex in the singular), which may be defined for the domain-specific integration that the chatbot or application is being used. For example, U.S. patent numbers are typically straightforward, having up to 8 digits, and sometimes are formatted with commas every three digits but often just a straight sequence of numbers; a regex for matching a U.S. patent number may be “\b\d{1,8}\b” in one embodiment, for example (note: the double quotes in the previous regex example are for identifying the bounds of the regex example and are not included in or a part of the regex). Of course, other regex statements may be defined and used, and these may be defined for the particular use case of the message generation system. In embodiments, when a match of a regex is found, the text is cleaned and stored as the document identifier or number (e.g., a patent number example would be “10123456” for U.S. Patent No.10,123,456) and the type of document / resource is inferred; for example, when a match of the regex defined for the U.S. patent numbers is found, it may be inferred that the document or resource type is a patent, particularly a U.S. patent unless a separate country code was provided and detected. Additionally, as will be discussed more below, a generic (or non-resource specific) resource type is defined as one of the predetermined resource types and this may be used for user-uploaded documents and / or other content that does not fit into one of the other predetermined resource types, for example, such as a catch-all.

[0258] In addition to or in lieu of identifying document resource identifiers from the user message, each of which isassociated with a predefined document or resource, such as a public document (e.g., patent, pre-grant patent application publication) or a user-uploaded document, specific portions of those documents or resources may be identified in this step. For example, a regex is predefined for one or more predetermined resource types, such as one or more anticipated resource types to be used by the domain-specific chatbot (e.g., U.S. patents, pre-grant patent application publications), as discussed above, but certain key terms or known phrases that may be used may be defined, each as a content identifier indicator. For example, for U.S. pre-grant patent application publications, the term “paragraphs” or the presence of square brackets around a four digit number may be used as a content identifier indicator. In embodiments, when the user message includes a content identifier indicator and a content identifier, the predefined regex(es) may be used to identify the resource type (e.g., here it is inferred from a regex match) and then extract the content identifier from the text. The content identifier indicator may be a plaintext name for the resource, such as “patent”, or may be another plaintext key phrase, term, character pattern or archetype, word, or other indicator, for example. For example, the phrase “summarize paragraph 1” may use “paragraph” as a content identifier indicator and “1” as the content identifier. Based on the chat context, such as the last discussed or loaded resource, it may be inferred the user is referring to this last resource and so the paragraph 1 of that resource is identified in the context data as updated. The method 1800 continues to step 1830.

[0259] In step 1830, augmentative data is obtained based on the identified context. For example, chat context that isdetermined in the previous step may be a dictionary (a Python dict) and may have attributes that correspond to resource types, and those attributes may then include resource and / or content identifiers. In one embodiment, the content is obtained using the content identifier, which may include a document or resource identifier and a section identifier that identifies a section or location within the document or resource where specific content that is only a portion and not the whole of the document or resource. For example, a paragraph number and pre-grant publication of a patent application may be used to identify a portion (the paragraph) of the patent application. In some embodiments, content may be needed that is not already obtained by the system and so a content fetching and ingestion process may be performed, such as one corresponding to the method of FIG.19, which is discussed below. After the augmentative data is obtained, the method 1800 continues to step 1840.

[0260] In step 1840, a response message is generated based on the user message and the augmentative data. Theresponse message is generated using a LLM message generation engine or system, such as the system 12 (FIG.1) and / or the engine 1712 (FIG.17). The LLM 119 may be an LLM hosted by a third party, such as OpenAI’s products (e.g., its “gpt” line of products, such as gpt-3.5-turbo and gpt-4) offered via their API, or the LLM 119 may be executed by the message generation system itself. The discussion above of steps 240 and 250 is incorporated herein and attributed to the step 1840 to the extent that discussion is not inconsistent with the discussion of the step 1840. The method 1840 continues to step 1850.

[0261] In step 1850, it is determined whether the response is suitable or complete and, if so, the method 1800 proceedsto step 1860 where the response message is sent or otherwise provided to the user device 14; otherwise, when it is determined that the response is not suitable or complete, the method 1800 proceeds to step 1870.

[0262] In one embodiment, the determination of step 1850 is made by determining whether a key term, phrase, orother indicator is within the response message returned by the LLM (or message generation network). For example, in one embodiment, the prompt input as a part of the language model input into the LLM includes an instruction to respond by only providing a response string of characters formed as a JSON (JavaScript Object Notation). For example, once the response message is received, an attempt to load the response string into a JSON is performed, such as through using “json.loads(response_message)” where “response_message” is the response message and “json” is the standard json Python library importing using “import json”. When there is a parsing error, it is determined that the response message is not formatted as a JSON and, accordingly, no more content is needed—accordingly, in such an example, it may be inferred from this that the response is suitable.

[0263] In step 1870, updated context data is determined based on augmentative data obtainment data in that itindicates which augmentative data to obtain for a next inference pass for message generation. As discussed above, in at least one embodiment, the message generation network is prompted to return a string formatted as a JSON. The string is then loaded into a dictionary using “json.loads”, and attributes of the dictionary may then be accessed. The dictionary, which is obtained from the JSON string of the response message, may have a predefined schema or configuration, such as where attribute names are defined along with value types or examples. For example, the prompt passed to the LLM may include a list of attribute names, a description orindication as to what they mean, suitable value types (e.g., string, array, int), and / or examples of response messages consisting of a string formed as a JSON. In embodiments, the same function used for determining the context in step 1820 is used to determine the updated context data based on the augmentative data obtainment data.

[0264] The method 1800 then proceeds back to step 1830. The method 1800 may have a predetermined number ofiterations (referred to also as a predetermined maximum) before determining to simply send a message to the user device instead of attempting to obtain content and generate another message. This mechanism may prevent an endless or long loop. In embodiments where the predetermined maximum is met, the method 1800 may perform one last pass of the message generation network with a clear and definite instruction to return a user-friendly message and not a JSON string. In embodiments, during this last pass, the instructions pertaining to the JSON may be omitted along with any other instructions pertaining to requesting more information. Once the response message is sent in step 1860, the method 1800 then ends.

[0265] With reference to FIG. 19, there is shown a method 1900 of ingesting content and generating content glossdata for the ingested content, according to one embodiment. The method 1900 may be used as a part of the method 1800, as will be described below.

[0266] The method 1900 begins with step 1910, wherein a determination is made as to whether content is needed tobe obtained by the system so that augmentative data, which may be portions thereof, may be retrieved and used for augmenting the language model input. This may be made according to the discussion above in the method 1800, particularly where context data determined from the user message (and / or indicated by augmentative data obtainment data) indicates certain content is needed and the system determines that that needed information is already obtained by the system. For example, when a regex detects a patent number in a user message, a data store of the system 12 that includes patent information is queried and, if a result is not found, it is determined the patent indicated by the patent number is new and information for that patent needs to be obtained; accordingly, the method 1900 would proceed to step 1930; otherwise, when it is determined that no new information (that is, information not yet obtained by the system (such as stored at a data store of the system and that needed data is only stored or made available by a third party, such as through a web- based API)) is needed, the method 1900 continues to step 1920. In step 1920, a response message is generated, such as through use of the method 1800.

[0267] In step 1930, a query is generated in order to fetch the content needed. As discussed above, it may bedetermined that content is needed in step 1910 when the user is inquiring about something that pertains to a document or other resource that is available from a third party API or other platform or system, but that is not yet stored in the data store(s) of the LLM message generation system. For example, when a new patent number is mentioned by a user in the user message, then the patent examination dataset (PED) API offered by the United States Patent and Trademark Office (USPTO) may be queried for a “patentNumber” equal to the extracted patent number from the user message. Of course, this is but one example that pertains to a patent- related chatbot application, as a variety of other types of documents and sources may be used.

[0268] Once the content is fetched, it is stored in a data store, such as a relational database such as SQL, DynamoDBor other NoSQL database, data lake, or other data store, which stores plaintext and may store data in other forms, such as images. In some embodiments, the retrieved content, which may correspond to a document, is then divided into sections, such as based on headers detected or inferred from the document or based on labels or other character delimiters such as paragraph or newline characters. For example, a patent document may be broken down into a plurality of entries, one for each column and line number pair, which is a common way of identifying particular portions of the patent. Another example is using paragraph numbers of a pre-grant patent application publication as sections, which may be referred to or identified by number. This enables certain sections to be looked up readily when detected in the context, such as where a particular paragraph, column / line number, claim, or other section of the document is inquired about. Once the query is generated, it may then be executed, such as by the data store API 1714, for example. The query results may then be processed, which may depend on the type of data retrieved and the form desired, for example. The method 1900 continues to step 1940.

[0269] In step 1940, content gloss data is generated for the content. As used herein, “content gloss data,” when usedin connection with content, refers to textual gloss data for the content. The textual gloss data discussed above in connection with step 1630 is data that is stored in a vector database and recalled based on queries at runtime, such as a neartext query, bm25, or other semantic search between the user message, a portion thereof, or other concepts derived from the user message or context of the chat. The content gloss data in this step is generated, at least in one embodiment, through splitting the content into a plurality of entries and generating a plurality of content gloss data entries for the content, with each content gloss data being for a particular portion of the content, which is referred to herein as a “section”. In one embodiment, a content gloss data entry, which is a textual gloss data entry for a section of content, is generated by passing the section of content into along with a textual gloss generation prompt into the message generation network 118.

[0270] The textual gloss generation prompt is a natural language text representation of words or other strings ofcharacters that instruct or otherwise direct the message generation network to generate a textual gloss for the content being passed in. In order to determine the textual gloss prompt that is suitable for the application in which the method is being practiced, written draft iterations of the textual gloss prompt may be had and tested through manually evaluating outputs relative to the inputs. This manual evaluation is for evaluating different versions of the textual gloss generation prompt in order to narrow down which language or strings for the prompt work best for the given use case in which the generated textual glosses are to be used. For example, in at least one embodiment, the textual glosses are to be used for semantic searching against a user message or input in order to identify sections of content that are relevant to the user message or input.

[0271] The textual glosses are automatically generated using an LLM, particularly, through using an instruction-basedLLM that is provided a language model input comprising the textual gloss generation prompt and the section of content for which the textual gloss is to be generated—this language model input is also referred to as a textual gloss generation language model input. It has been discovered that the state of the art (SOTA) LLM’s, particularly those pretrained for instruction-based text-to-text generation, such as OpenAI™ gpt-3.5-turbo, gpt-4, gpt-4.5-turbo, Mistral™ Chat (Le Chat) models such as the Mistral Large Model, Perplexity™ AI models, GroqChat™ models, Chatsonic™ models, Google™ Gemini models, Meta AI Llama models such as Llama 2, for example.

[0272] Previous SOTA LLMs supported context windows of only 4k, 8k, or 16k, and these limits oftentimes makeprocessing large amounts of text challenging or requiring many iterations. Recently, LLMs have been developed that have a much larger context window, particularly OpenAI’s 128k token gpt-4-turbo models, including “gpt-4-0125-preview”, “gpt-4-1106-preview”, and “gpt-4-1106-vision-preview”. For example, the vision-enabled LLMs, such as the gpt-4-1105-vision-preview, for example, may be used to process images as input along with text. Accordingly, text gloss data may be generated for images, such as for figures of a patent specification or graphs in a white paper, for example. In some embodiments, multiple content gloss data entries and the textual gloss generation prompt are passed as the language model input into the message generation network, which may use a large context window LLM, which is an LLM with a context window of at least 64k, or may use a very large context window LLM, which is an LLM with a context window of at least 128k, such as the OpenAI models listed above. The textual gloss generation prompt that is passed with a plurality of content gloss data entries may be a multi-entry textual gloss generation prompt that is a textual gloss generation prompt specifically configured to provide an instruction to process multiple content gloss data entries and generate a corresponding output for each. In embodiments, the textual gloss generation prompt, including the multi-entry textual gloss generation prompt, provide an instruction to generate an output that is structured according to a particular output structure, such as a particular data serialization, for example, JSON, XML, etc. In one embodiment, the predefined output structure used for the textual gloss generation prompt is JSON and, when used for the multi-entry textual gloss generation prompt, the predefined output structure is a JSON string representing an array of dictionaries, for example. One or more examples of such an output array or other serialized structured string output may be provided in the language model input in order to more confidently direct the LLM to produce an output structure appropriately.

[0273] Once the response of the message generation network is received, an attempt to parse the response messageis made, where the parsing attempt (and, if successful, parsing) is performed according to the predefined output structure (or predefined data serialization) used in the textual gloss generation prompt. For example, where the predefined output structure is JSON and when using Python, “json.loads(response_message)” may be called where “json” is importing from the Python json library through “import json” and the response_message is the response message generated by the message generation network based on the textual gloss generation language model input. As a result of successful parsing, a textual gloss for each section of content (or each textual gloss data entry) is obtained. The method 1900 proceeds to step 1950.

[0274] In step 1950, the content gloss data is stored in a vector data store. For example, as mentioned above, thecontent gloss data may be used for semantic searching against user messages received in real-time in a chatbot application, for example, and a response message to the user that is timely, accurate, and precise is desired. Therefore, the textual glosses may be generated with the notion of their intended use, and the textual gloss generation prompt configured accordingly. The content gloss data, which includes a generated textualgloss for a section of content and a content identifier for the section of content, is stored in a vector data store, such as Weaviate™ or Pinecone™. In some embodiments, the content gloss data may also be stored in a non-vector data store, such as DynamoDB. The method 1900 proceeds to step 1960.

[0275] In step 1960, a status message is sent to the user device, where the status message indicates a status of thenew content ingestion process, such as whether the process is still in progress, is complete, and / or additional information, such as a success / fail status or indication. The method 1900 ends.

[0276] According to embodiments, the above-discussed technology enables textual glosses to be automaticallygenerated and tailored specifically to a particular domain or field and / or to a particular intended use case so that the generated textual glosses represent a concise and salient representation of the section of content to which it pertains (on which it was generated) so that limitations of current vector database or data store systems are mitigated. For example, storing full text of sections of content often results in unnecessary processing and undesirable results in that, in certain fields, content to be stored for semantic searching may be wordy, abstracted, or not entirely straightforward. Simply vectorizing this data and using that vectorization for semantic searching, thus, results in undesirable results as portions, or even points made, in the content may not be relevant (and, indeed, may be dilutive of content that is actually relevant) and / or relevant points being made in the content may not be express, such as where they are only suggested. By providing an automatic textual gloss generation process that utilizes SOTA LLMs with a custom text gloss generation prompt, the system and method provided herein enables improved vector data store searching through what may be termed herein as “densified content vectorization” in that textual glosses of the section of content are vectorized instead of the entire section of content.

[0277] With reference to FIG. 20, there is shown a flowchart illustrating a method 2000 of generating a responsemessage in response to a user message, and this method 2000 particularly illustrates certain processing that may be used for constructing a language model input that includes augmentative data and the user message, which is then used as input into a message generation network for purposes of generating a response message that is then sent to the user device.

[0278] The method 2000 begins with step 2010, wherein a user message is received. This step is analogous to the step2010 of the method 1600 (FIG.16) and the step 1810 of the method 1800 (FIG.18). The discussions of the step 2010 and the step 1810 are hereby incorporated and attributed to the step 2010 to the extent those discussions are not inconsistent with the discussion of the step 1010. The method 2000 proceeds to step 2020.

[0279] In step 2020, context data is obtained. This step is analogous to step 1820 of the method 1800 (FIG.18) and thediscussion of the step 1820 is hereby incorporated and attributed to the step 2020 to the extent that discussion is not inconsistent with the discussion of the step 2020.

[0280] An example of a context data object having context data is shown below for a use case of the present methodand system in which a chatbot is being provided to a user at the user device.

[0281] The example JSON context object in the previous paragraph includes a “resources” object which providesinformation as to the needed / requested context for one or more user-inputted resources, such as those U.S. patents or pre-grant publications mentioned in a chat session with the patent-related chatbot as well as documents or resources uploaded by a particular user. Furthermore, the example JSON context object further provides information as to the needed / requested context for one or more predefined resources, which are those resources that are preloaded into the system, such as into a data store of the system, such as documents or resources that are applicable across an entire domain; for example, the Manual of Patent Examining Procedure (MPEP) published by the USPTO, 37 CFR (Code of Federal Regulations), 35 USC (United States Code), other patent guidelines, practice tips, cases, etc., as well as other documents broadly applicable may also be available and indicated in the context data.

[0282] The context data may indicate particular sections of content to obtain or may indicate a query to be executed.For example, the example JSON context object above includes three resource objects within the resources array, and each specifies one or more paragraphs and / or claims of the respective resource to obtain; for example, paragraph 10 and claims 1 and 8 are obtained for U.S. Patent No.10,112,332. Furthermore, one or more terms, or other input or data indicating an input / parameter(s), for a query may be included. For example, the resource with resource_id 10112332 has a “terms” array with two entries. The system and method of the present embodiment are configured to execute a semantic search, such as one performed by Weaviate usingOpenAI gpt-4 embeddings, using each of the terms against textual glosses for sections of content related to the resource 10112332, such as the specification of that patent. The method 2000 proceeds to step 2030.

[0283] In step 2030, content to be used as augmentative data is obtained based on the context data. The content isobtained from appropriate data stores as indicated by the context data. For example, the full text of paragraphs or defined sections (e.g., claims or abstract for patent documents) of a publication may be stored in a relational or other suitable database, such as DynamoDB, or example. Vector data stores may also be queried according to the context data, as discussed above.

[0284] For example, with reference to FIG.21, there is shown a method 2100 for obtaining content to be included ina language model input, and this method 2100 may be used for the step 2030, at least in one embodiment. The method 2100 begins with steps 2110 and 2120, which may be performed concurrently or serially, according to embodiments.

[0285] In step 2110, the context data object is parsed in order to identify a vector data store query, such as queryinga vector database of textual gloss entries for each section of a document (such as for each paragraph) with one or more terms or textual query inputs provided by the context data. In step 2120, content identifiers, such as paragraph or claim numbers, are identified based on the context data object. The method 2100 continues to step 2130.

[0286] In step 2130, the content is obtained using any vector data store queries and any content data store queries.These queries may be performed independently and / or concurrently, or may be performed in serial. The method 2100 ends. With reference back to FIG.20, once the content is obtained, which may include various sections of the content of a document or resource but not the whole thing, the method 2000 continues to step 2040.

[0287] In step 2040, augmentative data is constructed based on the obtained content. In embodiments, the contentobtained is processed into a language model input along with a predefined prompt and the user message. The content obtained may be labeled according to what it is, such as “Claims of the Patent” for claims, and used as or as a part of the augmentative data so that the LLM is aware of the relevancy of portions of content passed as a part of the language model input. The method 2000 continues to step 2050.

[0288] In step 2050, a response message is generated based on the language model input, which includes the usermessage and the augmentative data. This step is analogous to the step 1840 of the method 1800 (FIG.18) and that discussion is hereby incorporated and attributed to the step 2050 to the extent that discussion is not inconsistent with the discussion of the step 2050. The method 2000 proceeds to step 2060.

[0289] In step 2060, the response message is then provided to the user device. This step is analogous to the step1860 of the method 1800 (FIG.18) and that discussion is hereby incorporated and attributed to the step 2060 to the extent that discussion is not inconsistent with the discussion of the step 2060. The method 2000 then ends.

[0290] As used herein, a “data store” refers to any store for data and may be implemented using cloud storage, localstorage, or other storage implemented through hardware having non-volatile memory for storing data; and it isfurther noted this encompasses a variety of storage systems including databases, data warehouses, data lakes, and blob storage services, for example.

[0291] It is to be understood that the foregoing description is of one or more embodiments of the invention. Theinvention is not limited to the particular embodiment(s) disclosed herein, but rather is defined solely by the claims below. Furthermore, the statements contained in the foregoing description relate to the disclosed embodiment(s) and are not to be construed as limitations on the scope of the invention or on the definition of terms used in the claims, except where a term or phrase is expressly defined above. Various other embodiments and various changes and modifications to the disclosed embodiment(s) will become apparent to those skilled in the art.

[0292] As used in this specification and claims, the terms “e.g.,” “for example,” “for instance,” “such as,” and “like,” andthe verbs “comprising,” “having,” “including,” and their other verb forms, when used in conjunction with a listing of one or more components or other items, are each to be construed as open-ended, meaning that the listing is not to be considered as excluding other, additional components or items. Other terms are to be construed using their broadest reasonable meaning unless they are used in a context that requires a different interpretation. In addition, the term “and / or” is to be construed as an inclusive OR. Therefore, for example, the phrase “A, B, and / or C” is to be interpreted as covering all of the following: “A”; “B”; “C”; “A and B”; “A and C”; “B and C”; and “A, B, and C.” RECITAL OF INVENTIVE ASPECTS

[0293] 1. A method of generating a response message using a large language model (LLM), wherein themethod comprises: obtaining a user audio message for a conversation session; obtaining conversation history data for the conversation session; generating an inference model input based on the user audio message and the conversation history data; and generating a response message based on an inference output that is generated by inputting the inference model input into a low-rank model and a base model, wherein the inference model input is used as input into the low-rank model and the base model.

[0294] 2. The method of inventive aspect 1, wherein at least part of the user audio message is tokenized intowords conveying a user message.

[0295] 3. The method of inventive aspect 1, wherein at least part of the user audio message is tokenized intoone or more audible feature tokens, and wherein each of the one or more audible feature tokens is a token that is defined for an audio feature.

[0296] 4. The method of inventive aspect 3, wherein each of the audible feature tokens is determined using aclassifier that classifies portions of the user audio message.

[0297] 5. The method of inventive aspect 4, wherein a first one of the audible feature tokens indicates a tone ofthe user audio message.

[0298] 6. The method of inventive aspect 4, wherein a first one of the audible feature tokens indicates anemotion conveyed by the user audio message.

[0299] 7. The method of inventive aspect 1, wherein at least part of the user audio message is tokenized intowords conveying a user message, wherein at least part of the user audio message is tokenized into one or more audible feature tokens, and wherein each of the one or more audible feature tokens is a token that is defined for an audio feature.

[0300] 8. A method for operating a neural network, comprising:training a transformer language model using an input embedding having at least one audio feature token; obtaining a user audio message for a conversation session; generating an inference model input based on the user audio message, wherein the inference model input includes an audio feature; and generating a response message based on an inference output that is generated by inputting the inference model input into the trained transformer language model.

[0301] 9. The method of inventive aspect 8, wherein positional encoding data is generated for the at least oneaudio feature token, and wherein the positional encoding data specifies a location at which the audio feature token pertains.

[0302] 10. A method of generating an output using a dual-selectable ensemble network, comprising the steps of:selecting at least one of a first neural network model and a second neural network model based on runtime information; inputting a message into the first neural network model and a base model to generate a backbone output; and generating a message generation network output as a result of a forward pass through the second neural network with the backbone output being used as input into the second neural network.

[0303] 11. A method of generating an output using a dual-selectable ensemble network, comprising the steps of:selecting a first neural network model based on user data; selecting a second neural network model based on runtime information;inputting a message into the first neural network model and a base model to generate a backbone output; and generating a message generation network output as a result of a forward pass through the second neural network with the backbone output being used as input into the second neural network.

[0304] 12. A method of generating an output using a dual-selectable ensemble network, comprising the steps of:selecting a low-rank model based on a user; selecting a head network based on runtime information; inputting a message into the low-rank model and a base model to generate a backbone output; and generating a message generation network output as a result of a forward pass through the head network with the backbone output being used as input into the head network.

[0305] 13. A method of generating an output using a dual-selectable ensemble network, comprising the steps of:a base model based on the custom user data, wherein the base model is a large language model (LLM); generating custom training data based on the custom user data and the selected base model; and training a low-rank model using the base model and the custom training data to obtain a custom low-rank model.

[0306] 14. The method of inventive aspect 13, wherein the custom training data is generated using a predefinedprompt structure for the base model.

[0307] 15. A method of training a low-rank model using custom training data, comprising the steps of:receiving custom user data from a custom data party; generating custom training data based on the received custom user data, wherein the custom training data is generated according to a predefined prompt structure used for a base model; training a low-rank model using a low-rank adaptation technique, wherein the low-rank model is trained in parallel with the base model using the custom user data to generate a custom low-rank adapter; receiving a user message from a user; and generating a response message through use of the custom low-rank adapter and the base model.

[0308] 16. The method of inventive aspect 15, wherein the base model is an application-specific base model.

[0309] 17. The method of inventive aspect 16, wherein the application-specific base model is one of a pluralityof application-specific base models that are each generated by training the LLM according to a different predefined prompt structure, and wherein the application-specific base model is selected from the plurality of application-specific base models for being used as a part of generating the low-rank model and is selected according to the custom user data and / or the custom data party.

[0310] 18. The method of inventive aspect 17, wherein the custom training data is generated according to thepredefined prompt structure based on using the custom user data and predefined custom training template data.

[0311] 19. The method of inventive aspect 18, wherein the predefined custom training template data and thecustom user data are passed as input into a large language model (LLM) in order to generate the custom training data.

[0312] 20. A method of autonomously training a message generation network, comprising the steps of training alarge language model (LLM) using non-speech audio feature tokens.

[0313] 21. The method of inventive aspect 20, wherein the non-speech audio feature tokens are a part of apredefined token set.

[0314] 22. The method of inventive aspect 21, wherein the non-speech audio feature tokens include at least onetoken that is defined for an emotion.

[0315] 23. The method of inventive aspect 20, further comprising obtaining a model input for the LLM based onusing an audio classification technique to determine a token for a non-speech audio feature.

[0316] 24. The method of inventive aspect 23, wherein the audio classification technique is a speech emotionrecognition technique that is used to determine an emotion based on audio data.

[0317] 25. A method of generating a response message for a user message, comprising the steps of:receiving a user message from a user; extracting one or more topics from the user message; obtaining a subject based on the one or more topics, wherein the subject and the one or more topics are associated with one another as topic-subject data; generating an intermediate target related to the user message, the topic, and / or the subject; generating a prompt based on the intermediate target; and generating a response message based on a forward pass of a message generation network that takes, as input, the generated prompt.

[0318] 26. The method of inventive aspect 25, wherein the intermediate target is generated based on a targetsubject and the user message.

[0319] 27. The method of inventive aspect 25, wherein an autoregressive target generation network is carriedout to generate a plurality of intermediate targets including the intermediate target that is determined based on a source and a target.

[0320] 28. The method of inventive aspect 27, wherein the plurality of intermediate targets are serialized to forma target pathway that begins at the source and ends at the target, interpolated therebetween with one or more of the plurality of intermediate targets.

[0321] 29. The method of inventive aspect 27, wherein the autoregressive target generation network prioritizesgenerating intermediate targets closer in the target pathway to the source.

[0322] 30. The method of inventive aspect 29, wherein an autoregressive target generation network is used togenerate the intermediate target related to the user message, the topic, and / or the subject, and wherein the intermediate target is used to generate the prompt.

[0323] 31. The method of inventive aspect 30, wherein the autoregressive target generation network is asemantic denominator extractor network that takes, as two inputs, and generates, as output, a semantic denominator that represents a semantic denominator of the two inputs and, when using the source and the target as the two inputs, the semantic denominator of the source and the target is generated.

[0324] 32. The method of inventive aspect 25, wherein user knowledge data pertaining to the user is obtainedfrom a user knowledge data store, wherein the user knowledge data store includes user knowledge data entries generated based on extracted knowledge that is extracted from user message data using a knowledge extractor network, wherein the user knowledge data indicates a topic and a sentiment toward the topic, and wherein the user knowledge data is used for generating the prompt.

[0325] 33. The method of inventive aspect 32, wherein the user knowledge data is used to inform the subjectthat is used for generating the prompt based on the sentiment toward the topic.

[0326] 34. A method of introducing an intermediate target into a conversation, comprising:determining a source based on a context of a conversation with a user; determining a target; generating an intermediate target based on the source and the target through use of an autoregressive target generation network; generating an input for a message generation network based on the intermediate target; and generating a message based on an output that is produced by inputting the generated input into the message generation network.

[0327] 35. The method of inventive aspect 34, wherein the source refers to the context of the conversation whenthe user message is received, and wherein the target refers to a target context for the conversation, and wherein the target and the target context are based on the context of the conversation with the user.

[0328] 36. The method of inventive aspect 35, wherein the autoregressive target generation network is semanticdenominator extractor network that takes, as two inputs, and generates, as output, a semantic denominator that represents a semantic denominator of the two inputs and, when using the source and the target as the two inputs, the semantic denominator of the source and the target is generated.

[0329] 37. The method of inventive aspect 36, wherein the user knowledge data is used to inform selection ofthe intermediate target.

[0330] 38. The method of inventive aspect 37, wherein a plurality of candidate intermediate targets are generatedfor a set of inputs, wherein one of the candidate intermediate targets is selected as the intermediate target, and wherein the user knowledge data is used to inform selection of the candidate intermediate target to be the intermediate target.

[0331] 39. The method of inventive aspect 36, wherein the autoregressive target generation network is carriedout to generate a plurality of intermediate targets, and wherein the plurality of intermediate targets includes the intermediate target as a first-level target and second-level intermediate targets that are generated based on the intermediate target, the source, and the target.

[0332] 40. The method of inventive aspect 39, wherein the plurality of intermediate targets are filtered to removeresults that are determined to be above a predetermined similarity threshold to other results.

[0333] 41. The method of inventive aspect 39, wherein the plurality of intermediate targets are serialized to forma target pathway that begins at the source and ends at the target, interpolated therebetween with one or more of the plurality of intermediate targets.

[0334] 42. The method of inventive aspect 41, wherein the plurality of intermediate targets are generated inresponse to receiving the user message.

[0335] 43. The method of inventive aspect 42, wherein the autoregressive target generation network prioritizesgenerating intermediate targets closer in the target pathway to the source.

[0336] 44. The method of inventive aspect 36, further comprising: extracting one or more topics from the usermessage; and obtaining a subject based on the one or more topics, wherein the subject and the one or more topics are associated with one another as topic-subject data, wherein the prompt is generated based on the subject.

[0337] 45. The method of inventive aspect 34, wherein the autoregressive target generation network isimplemented using a large language model (LLM) that takes, as input, data pertaining to the target and the source.

[0338] 46. The method of inventive aspect 45, wherein user knowledge data pertaining to the user is obtainedfrom a user knowledge data store, wherein the user knowledge data store includes user knowledge data entries generated based on extracted knowledge that is extracted from user message data using a knowledge extractor network, wherein the user knowledge data indicates a topic and a sentiment toward the topic, and wherein the user knowledge data is used for generating the prompt.

[0339] 47. The method of inventive aspect 46, wherein the user knowledge data is used to inform the target thatis used for generating the prompt based on the sentiment toward the topic.

[0340] 48. A method of operating a message generation network in order to generate response messages thatform a part of a conversation with a user, comprising: determining a source input based on a context of a conversation with a user; determining a target input representing a target subject; generating an intermediate target based on the source input and the target input through use of an autoregressive target generation network; generating an input for a message generation network based on the intermediate target; and generating a message based on an output that is produced by inputting the generated input into the message generation network.

[0341] 49. A computer system having:a message generation network configured to generate a response message for a user message as a part of a conversation with a user; and an autoregressive target generation network configured to generate an intermediate target based on a target subject and a context of the conversation with the user, wherein the response message is generated based on the intermediate target.

[0342] 50. The computer system of inventive aspect 49, wherein the message generation network is a pretrainedLLM.

[0343] 51. A method of operating a message generation network in order to generate response messages thatform a part of a conversation with a user, comprising: operating a message generation network in order to carry out an automated conversation with a user; generate an intermediate target based on a target subject and a context of the conversation with the user; andgenerating a response message using the message generation network based on the intermediate target, wherein the response message is a part of the automated conversation with the user.

[0344] 52. The method of inventive aspect 51, wherein the intermediate target is generated as a part of anintermediate target pathway generation process that generates a plurality of intermediate targets in order to form a serialized target pathway.

[0345] 53. A method of training a low-rank model using custom training data, comprising the steps of:receiving custom user data from a custom data party; selecting a base model based on the custom user data, wherein the base model is a large language model (LLM); generating custom training data based on the custom user data and the selected base model; and training a low-rank model using the base model and the custom training data to obtain a custom low-rank model.

[0346] 54. The method of inventive aspect 53, wherein the custom training data is generated using a predefinedprompt structure for the base model.

[0347] 55. A method of training a low-rank model using custom training data, comprising the steps of:fine-tuning a pretrained large language model (LLM) to generate an application-specific base model that follows a predefined prompt structure; receiving custom user data from a custom data party; generating custom training data according to the predefined prompt structure based on the received custom user data; training a low-rank model using a low-rank adaptation technique where the low-rank model is trained in parallel with the application-specific base model using the custom user data to generate a custom low-rank adapter; receiving a user message from a user; and generating a response message through use of the custom low-rank adapter and the application-specific base model.

[0348] 56. The method of inventive aspect 55, wherein the application-specific base model is one of a pluralityof application-specific base models that are each generated by training the LLM according to a different predefined prompt structure, and wherein the application-specific base model is selected from the plurality of application-specific base models for being used as a part of generating the low-rank adapter network and is selected according to the custom user data and / or the custom data party.

[0349] 57. The method of inventive aspect 56, wherein the custom training data is generated according to thepredefined prompt structure based on using the custom user data and predefined custom training template data.

[0350] 58. The method of inventive aspect 57, wherein the predefined custom training template data and thecustom user data are passed as input into a large language model (LLM) in order to generate the custom training data.

[0351] 59. An end-to-end autonomous message generation training and deployment system, comprising:a custom training data retrieval interface implemented through one or more computers configured to receive custom user data from a custom data party; a large language model (LLM) base model trainer implemented through one or more computers configured for generating an application-specific base model that follows a predefined prompt structure; a low-rank model trainer implemented through one or more computers configured for generating a custom-trained low-rank model based on using the custom user data and the application- specific base model; and a message generation inference generator implemented through one or more computers configured for generating a response message through use of the low-rank model and the application- specific base model.

[0352] 60. A method of autonomously training a message generation network, comprising the steps of:training a large language model (LLM) to generate an application-specific base model that follows a predefined prompt structure; receiving custom user data from a user or a third party; generating custom training data according to the predefined prompt structure based on the received custom user data; and generating a low-rank model based on training the application-specific base model using the custom user data.

[0353] 61. The method of inventive aspect 60, wherein training the LLM to generate the application-specificinstruct model is performed by fine-tuning the LLM according to the predefined prompt structure.

[0354] 62. The method of inventive aspect 61, wherein the application-specific instruct model is generated byfine-tuning the LLM using task-oriented fine-tuning.

[0355] 63. A method of training multiple low-rank models using shared training, comprising the steps of:obtaining a large language model (LLM);obtaining a plurality of training data sets, each training data set corresponding to one of a plurality of low-rank models to be trained; and training the plurality of low-rank models using the base model and the training data sets, wherein the training includes using an output from a forward pass of the base model for training each of the plurality of low-rank models.

[0356] 64. The method of inventive aspect 63, wherein the training includes: a shared training phase where theoutput from the forward pass of the base model for training each of the plurality of low-rank models; and an individualized phase where training is performed without sharing outputs from the base model.

Claims

CLAIMS 1. A method of generating a response to a user message, comprising: obtaining a text document specified by a user, wherein the text document is a document that has text; chunking the text document into a plurality of chunks; generating text gloss data for each chunk of the plurality of chunks based on inputting the chunk along with a predetermined text gloss generation prompt into a large language model; storing the text gloss data for each chunk of the plurality of chunks into a vector data store along with an identifier for the chunk and / or the chunk itself; receiving a user message; querying the vector data store with a vector data store query, wherein the vector data store query is generated based on the user message; obtaining one or more chunks based on an output resulting from querying the vector data store with the vector data store query; generating a language model input based on the one or more obtained chunks; and generating a response message based on inputting the language model input into the large language model or another large language model.

2. The method of claim 1, wherein, for each chunk of the plurality of chunks, a text gloss language model input is generated, and wherein the text gloss language model input includes the chunk and the predetermined text gloss generation prompt.

3. The method of claim 2, wherein, for each chunk of the plurality of chunks, the text gloss language model input further includes a portion of the text document that is different from the chunk and / or content based on the text document that is different from the chunk.

4. The method of claim 2, wherein, for each chunk of the plurality of chunks, the text gloss language model input includes a summary of the text document or text content from the text document that includes content of at least one other chunk than the chunk.

5. The method of claim 1, further comprising: obtaining an image from the text document;generating text gloss data for the image based on inputting the image into the large language model or another vision-enabled large language model; and storing the text gloss data for the image into the vector data store.

6. The method of claim 1, wherein the predetermined text gloss generation prompt includes text instructing the large language model to generate a structured output that contains a list of glossary terms or phrases relevant to subject content identified as the chunk.

7. The method of claim 6, wherein at least one vector data store entry for each chunk of the plurality of chunks is generated based on parsing the structured output to obtain the text gloss data for the chunk.

8. A method of generating a response to a user message, comprising: obtaining a text document specified by a user, wherein the text document is a document that has text; storing the text document in a content data store; chunking the text document into a plurality of chunks; generating text gloss data for each chunk of the plurality of chunks based on inputting the chunk along with a predetermined text gloss generation prompt into a large language model; storing the text gloss data for each chunk of the plurality of chunks into a vector data store along with an identifier for the chunk; receiving a user message; querying the vector data store with a vector data store query, wherein the vector data store query is generated based on the user message; obtaining one or more chunks based on one or more chunk identifiers returned from querying the vector data store with the vector data store query; generating a language model input based on the one or more obtained chunks; and generating a response message based on inputting the language model input into the large language model or another large language model.

9. The method of claim 8, wherein, for each chunk of the plurality of chunks, a text gloss language model input is generated, and wherein the text gloss language model input includes the chunk and the predetermined text gloss generation prompt.

10. The method of claim 9, wherein, for each chunk of the plurality of chunks, the text gloss language model input further includes a portion of the text document that is different from the chunk and / or content based on the text document that is different from the chunk.

11. The method of claim 9, wherein, for each chunk of the plurality of chunks, the text gloss language model input includes a summary of the text document or text content from the text document that includes content of at least one other chunk than the chunk.

12. The method of claim 8, further comprising: obtaining an image from the text document; generating text gloss data for the image based on inputting the image into the large language model or another vision-enabled large language model; and storing the text gloss data for the image into the vector data store.

13. The method of claim 8, wherein the predetermined text gloss generation prompt includes text instructing the large language model to generate a structured output that contains a list of glossary terms or phrases relevant to subject content identified as the chunk.

14. The method of claim 13, wherein at least one vector data store entry for each chunk of the plurality of chunks is generated based on parsing the structured output to obtain the text gloss data for the chunk.

15. A method of generating a response to a user message, comprising: obtaining a text document specified by a user, wherein the text document is a document that has text; storing the text document in a content data store; chunking the text document into a plurality of chunks; generating text gloss data for each chunk of the plurality of chunks based on inputting the chunk along with a predetermined text gloss generation prompt into a large language model; storing the text gloss data for each chunk of the plurality of chunks into a vector data store along with an identifier for the chunk; receiving a user message; generating a vector data store query based on an output generated by inputting the user message into the large language model or a second language model;querying the vector data store with the vector data store query; obtaining one or more chunks based on one or more chunk identifiers returned from querying the vector data store with the vector data store query; generating a language model input based on the one or more obtained chunks; and generating a response message based on inputting the language model input into the large language model or another large language model.

16. The method of claim 15, wherein the vector data store query is determined by parsing the user message to identify one or more text strings to be used for querying the vector data store.

17. The method of claim 15, further comprising: storing the text document into a content data store, wherein the one or more chunks are obtained by querying the content data store with the one or more chunk identifiers returned from querying the vector data store.

Images