Machine learning models using evidenced content lexical insertion

By introducing model plugins into generative machine learning models, utilizing source metadata identifiers from credible data sources and retrieving content from credible data sources, and directly inserting them into the output sequence, the efficiency and accuracy issues of generative models are solved, achieving more efficient and accurate information generation.

CN123125600APending Publication Date: 2026-11-17MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202580028041.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-12-02
Filing Date
2025-03-13
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Generative machine learning models have shortcomings in terms of efficiency and accuracy, especially in terms of high computational resource consumption, information accuracy and stability, and the range of training data is limited, making them unable to accurately respond to private or recent data.

Method used

By introducing model plugins into machine learning models, source metadata from credible data sources is used to identify credible content and retrieve the corresponding content from the credible data sources, which is then directly inserted into the output sequence, avoiding probability generation and improving accuracy and efficiency.

Benefits of technology

It improves the accuracy and efficiency of generative models, ensures the accuracy of evidence-based content, and reduces computational resource consumption, avoiding model illusions and training data limitations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123125600A_ABST
    Figure CN123125600A_ABST
Patent Text Reader

Abstract

A computing system (10) is provided that receives a lexicalized cue word (33) at a machine learning model (18), generates a model-generated content portion (40B) of an output sequence (38) of output lexicalized words (40) in response to the lexicalized cue word, and identifies source metadata (29) for a credible data source (44) in the model-generated content portion of the output sequence. When the source metadata is identified, the computing system at least temporarily suspends probabilistic generation of the output sequence using the machine learning model, retrieves credible content (52) from the credible data source using the source metadata, writes the output lexicalized words corresponding to the credible content into the credible content portion (40A1) of the output sequence, and transmits the output sequence to additional computing processes for, for example, display, storage, or additional downstream processing.
Need to check novelty before this filing date? Find Prior Art

Description

Background Technology

[0001] In recent years, generative machine learning models have achieved impressive results. These models have been applied to generative tasks across a wide range of fields, such as natural language generation, computational chemistry, image and video generation, and computer code generation. The largest generative models are capable of producing outputs that are very similar to human output and have achieved high scores on accuracy benchmarks for certain tasks. However, as discussed below, this accuracy comes at a cost and is not always achievable for all types of model interactions. Therefore, as these generative models continue to evolve, opportunities remain to improve their accuracy and efficiency. Summary of the Invention

[0002] A computing system is provided that receives a lexicalized prompt word at a machine learning model, generates a model-generated content portion of an output sequence of output lexicalized words in response to the prompt word, and identifies source metadata for a credible data source in the model-generated content portion of the output sequence. Upon identification of the source metadata, the computing system at least temporarily suspends the probabilistic generation of the output sequence word-by-word using the machine learning model, retrieves credible content from the credible data source using the source metadata, writes the output lexicalized words corresponding to the credible content into the credible content portion of the output sequence, and transmits the output sequence to an additional computing process for, for example, display, storage, or additional downstream processing.

[0003] This summary is provided to introduce, in a simplified form, some concepts that will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to the implementation of solutions to any or all of the shortcomings pointed out in any part of this disclosure. Attached Figure Description

[0004] Figure 1 A schematic diagram of a computing system configured according to a first example of this disclosure is shown, which implements a machine learning model configured to perform evidenced content lexical insertion during inference.

[0005] Figure 2 It shows Figure 1 The schematic diagram of the computing system shows a machine learning model that generates an output sequence of output tokens, and a post-processor for the model plugin that examines the output sequence during generation, determines that edible content should be written into the output sequence, retrieves edible content from the edible content source, and directly writes the output tokens for the edible content into the output sequence.

[0006] Figure 3A schematic diagram of a computing system configured according to a second example of this disclosure is shown, which implements a machine learning model configured to perform evidenced content lexical insertion during inference.

[0007] Figure 4 It shows Figure 3 A schematic diagram of the computing system shows example prompt words indicating a data source and a corresponding response, which has model-generated lexical units and data-content lexical units, the data-content lexical units being inserted by the data-content module of the preprocessor of the model plugin of the machine learning model.

[0008] Figure 5 It shows Figure 3 The diagram illustrates another example prompt word with a data source and a corresponding response, which has model-generated lexical units and data-content lexical units, the data-content lexical units being inserted by the data-content module of the post-processor of the model plugin of the machine learning model.

[0009] Figure 6 This is a flowchart of a computerized method for implementing a machine learning model that performs evidenced content lexical insertion, according to a first exemplary embodiment of the present disclosure.

[0010] Figure 7A This is a flowchart of a computerized method for implementing a machine learning model that performs evidenced content lexical insertion, according to a second exemplary embodiment of this disclosure.

[0011] Figure 7B yes Figure 7A The flowchart is a continuation of the computerized method.

[0012] Figure 7C yes Figure 7B The flowchart is a continuation of the computerized method.

[0013] Figure 8 This is a flowchart of a computerized method for implementing a machine learning model that performs evidenced content lexical insertion, according to a third exemplary embodiment of this disclosure.

[0014] Figure 9 It shows that it can be achieved Figures 1 to 5 A schematic diagram of an example computing environment for a computing system. Detailed Implementation

[0015] As mentioned above, generative machine learning models have evolved to the point where their outputs are remarkably similar to human output across many categories of tasks. For example, for pre-trained transformer-based language models, benchmark accuracy typically increases with the size of the parameter set, with the largest models currently exceeding hundreds of billions of parameters. At this scale, such models suffer from drawbacks in terms of efficiency and accuracy. In terms of efficiency, training and inference with such large models consumes significant computational resources, energy, and time. In terms of accuracy, the probabilistic nature of such models can lead to model illusions, where the model responds to cue words with inaccurate information not included in its training data. Furthermore, even in response to the same or similar inputs, the output of such models can vary, making them unstable and unusable in applications requiring reliable and stable outputs. Additionally, the range of training data for any generative model is limited. For example, training data may not include inaccessible private data or extremely recent data. Due to these limitations, generative models may respond to cue words inaccurately with stale or incorrect information.

[0016] One existing approach to addressing these issues is to use retrieval augmentation to augment the cue words provided to the language model. This retrieval augmentation retrieves information relevant to the cue word from a grounded, trusted data source. The information retrieved from the grounded data source is used to augment the cue word (e.g., by appending the retrieved information to the cue word), and the augmented cue word is sent to the language model to generate a response. The generative model generates a response based on both the original information contained in the augmented cue word and the information retrieved from the grounded data source, which is also contained in the augmented cue word.

[0017] One drawback of using evidence-based data sources for retrieval augmentation is that it leads to sending lengthy cue words to the generative model, increasing computational resources, energy, and time consumed during inference. Another drawback is that during model inference, information from evidence-based data sources is processed probabilistically as model input, thus there is no guarantee that the evidence-based content will appear accurately or reliably in the output.

[0018] To solve the above problems, such as Figure 1As shown, a computing system 10 is provided. The computing system 10 includes processing circuitry 12 and associated memory 14, which stores instructions 16 that, when executed, cause the processing circuitry 12 to perform the following functions: The processing circuitry 12 is configured to instantiate a trained machine learning model 18 and to instantiate a model plugin 20. The model plugin 20 is configured to provide an interface with the machine learning model 18 to enable user-defined functionality to be implemented at the machine learning model 18. The model plugin 20 may be provided as additional software installed in an existing machine learning model 18, or it may be incorporated into the machine learning model 18 as a native interface. The machine learning model 18 may include a transformer 18A. Therefore, the machine learning model 18 may be, for example, a generative transformer-based model, including an encoder-decoder architecture or a decoder-only architecture. The transformer-based machine learning model 18 may be unimodal or multimodal. As some examples, the input in a unimodal or multimodal configuration may include natural language input, image input, video input, audio waveform input, and / or parameterized data input from a data stream. As some specific examples, machine learning model 18 could be a generative large language model with billions of parameters, such as GPT-3.5, GPT-4o, ORCA-2, or LLaMA-2.

[0019] During training, a transformer 18A of the machine learning model 18 is trained on a training dataset 18T, which includes evidenced content 19 encoded with associated evidenced content source metadata 29. The evidenced content source metadata 29 may include a link 54, such as a URL, to an evidenced content source 44 where the evidenced content 19 can be accessed. For example, the source metadata 29 may be encoded in JSON format, which may include keys such as the title, author, and publication year identifying a public domain work or other copyrighted work. The source metadata 29 may also include some or all of the text, images, video, and / or audio associated with the evidenced content 19. Figure 2 In the example, an entry for Edgar Allan Poe's poem "The Raven" is discussed, which can be included in the training dataset 18T. By including source metadata 29 in the training dataset 18T, it should be understood that when a user queries specific credentialed content 19 during inference, the machine learning model 18 will be trained to output source metadata 29 for that credentialed content 19.

[0020] During inference, cue words 22 are received via cue word interface 24. In one set of examples, cue word interface 24 may be a graphical user interface of a program such as a chatbot, browser, or productivity application; in another example, cue word interface 24 may be an application programming interface. Cue words 22 consist of text data, which may include unstructured text such as natural language input. When using a multimodal model, cue words 22 may include other input modalities, such as images, videos, or audio. Cue words 22 are passed through lexicon 32 to generate lexiconized cue words 33, which include the input sequence 34 of input lexicons 36.

[0021] Processing circuit 12 is configured to receive a lexicalization cue word 33 at machine learning model 18. In response to receiving the lexicalization cue word, transformer 18A of machine learning model 18 generates a model-generated content portion of the output sequence 38 containing output lexical units 40, as shown in the figure. As shown in the word-by-word generation loop 39, the output sequence 38 is generated word-by-word, regressively generating one lexical unit at a time based on the lexicalization cue word 33 and the current state of the output sequence 38, until a termination condition is met.

[0022] During the execution of the word-by-word generation loop 39, the postprocessor 27 of the model plugin 20 for the machine learning model 18 is configured to examine the output words 40 in the output sequence 38 and identify the edible content source metadata words 40B1 (i.e., lexicalized source metadata 29) for the edible data source 44 in the model-generated content portion of the output sequence 38. The model-generated content portion includes the output words 40B generated by the model, and in these model-generated words 40B, the words that encode the edible content source metadata 29 are referred to as edible content source metadata words 40B1.

[0023] Upon identifying the data source metadata lexicon 40B1 via the post-processor, model plugin 20 is configured to at least temporarily halt the generation of word-by-word probabilistic probabilistics of the output sequence using machine learning model 18 in generation loop 39. Post-processor 27 instructs data source module 42 to retrieve data source content from data source 44 using source metadata 29 encoded in data source metadata lexicon 40B1. Typically, this interaction occurs via network 46, such as the Internet, but can also occur, for example, via a local area network. As described above, source metadata 29 may include a link 54 pointing to a location on data source 44 where data content 19 can be accessed and downloaded.

[0024] The evidenced content module 42 of model plugin 20 is configured to write evidenced content output terms 40A corresponding to evidenced content 19 retrieved from evidenced content source 44 into the evidenced content portion 40A1 of output sequence 38, thereby forming an updated output sequence 38U. For example, model plugin 20 is configured to transfer the updated output sequence 38U to additional computational processes, such as graphical user interface (GUI), downstream applications, or stored procedures.

[0025] As shown in the figure, the updated output sequence 38U can be de-lexicalized by lexer 32, generating a response 48. Response 48 includes text data comprising: model-generated content 50 based on the model-generated content portion (i.e., model-generated lexical 40B) of the updated output sequence 38U; evidenced content 52 based on evidenced content portion 40A1 of the updated output sequence 38U; and a link 54 to evidenced content 19 at evidenced data source 44, which can be encoded using evidenced content source metadata lexical 40B1. Source metadata 29 and 29A for evidenced content 52 and model-generated content 50 can be included in response 48, respectively. Unlike source metadata 29, source metadata 29A for model-generated content 50 includes information about its generation via machine learning model 18 and model plugin 20, but does not include information about evidenced content source 44, as model-generated content 50 is not retrieved from that source.

[0026] Response 48 can be output to prompt interface 24. Therefore, in one example, the aforementioned additional calculation process can be a graphical user interface (GUI), and processing circuitry 12 can be configured to transmit output sequence 38 for display at the GUI, where model-generated content 50 and data-driven content 52 are visually distinguishable. Text associated with data-driven content 52 can be marked at the GUI with indicators of data-driven data source 44 (e.g., a link 54 to data-driven data source 44). Examples are discussed below. Figure 2 As shown, the model-generated content 50 uses plain text, while the data-driven content 52 uses bold text. Of course, other forms of emphasis can also be used, such as color, size, font, outline, underline, highlight, quotations, etc. In another example, the prompt word interface 24 can be an API or a storage interface, and the additional calculation process can be a downstream calculation program, such as a prompt word API or storage interface.

[0027] The computing system 10 can be configured to insert non-training labels 31 into the updated output sequence 38U using the evidenced content lexical insertion technique described herein, thereby preventing or prohibiting the training of a third-party model based on the output of the machine learning model 18 and / or the evidenced data source 44. To this end, the processing circuit 12 can be configured to: label the evidenced content 52 with source metadata 29 as first source metadata, and / or label the model-generated content 50 with second source metadata 29A indicating the output generated by the machine learning model. The processing circuit 12 can also be configured to exclude the updated output sequence 38U and response 48 from the training corpus of the additional machine learning model, at least in part, based on determining that the output sequence is labeled with the first source metadata 29 or the second source metadata 29A. This can be achieved by inserting non-training labels 31 associated with the model-generated content 50 and / or the evidenced content 52 into the training data as appropriate. In this way, the output of the machine learning model 18 and / or the output of the evidenced data source 44 can be avoided when training the additional machine learning model.

[0028] Now go to Figure 2 , will describe Figure 1 Example use case scenarios for computing system 10. Due to Figure 2 Components and Figure 1Similarly, these components will not be redescribed except where used to illustrate this use case. As shown in the figure, a prompt word 22 is received, which requests the machine learning model 18 to explain who wrote the famous poem "The Raven" and state the first stanza of the poem. The machine learning model 18 receives the lexicalized prompt word 33 and begins to generate the output sequence 38 word-for-word. The first sentence generated by the machine learning model states: "Edgar Allan Poe wrote 'The Raven.' The first stanza is as follows:." Since this text is generated by the model and does not exist in the training data for the poem "The Raven" itself, the text does not contain source metadata and is therefore generated probabilistically following the word-for-word generation loop 39. Next, the machine learning model 18 begins to output source metadata 29 encoded in the data content source metadata lexical 40C (e.g., {"Title":"The Raven", "URI:example_uri / path#parameters"}), at which point the post-processor 27 identifies the source metadata 29 of the data content 52 and passes it to the data content module 42. Evidenced content module 42 uses the link in lexicographical unit 40C to download evidenced content 19 (the first section of "The Raven") from evidenced content source 44 and displays it in response 48. In this way, the response includes both content probabilistically generated from model 18 and deterministically generated content from the evidenced content source. As used herein, "probabilistically generated content" refers to content generated using a generation loop of a trained generative machine learning model that predicts the output sequence of content probabilistically; while "deterministically generated content" refers to content that is deterministically retrieved and directly written into the output sequence, and is not probabilistically generated. This ensures the accuracy of the content downloaded from evidenced content source 44, while also improving model efficiency by avoiding invoking machine learning model 18 to generate the text of the first section of the poem.

[0029] Now go to Figure 3This illustration shows a second configuration of a computing system 10A according to the present disclosure. The computing system 10A includes processing circuitry 12 and associated memory 14 storing instructions 16 that, when executed, cause the processing circuitry 12 to perform the following functions: The processing circuitry 12 is configured to instantiate a trained machine learning model 18 and to instantiate a model plugin 20. The model plugin 20 is configured to provide an interface with the machine learning model 18 to enable user-defined functionality to be implemented at the machine learning model 18. The model plugin 20 may be provided as additional software installed in an existing machine learning model 18, or it may be incorporated into the machine learning model 18 as a native interface. The machine learning model 18 may be a generative transformer-based model, including encoder-decoder, encoder-only, or decoder-only architectures. The transformer-based machine learning model 18 may be unimodal or multimodal. As some examples, the input in a unimodal or multimodal configuration may include natural language input, image input, video input, audio waveform input, and / or parameterized data input from a data stream. As some specific examples, machine learning model 18 could be a generative large language model with billions of parameters, such as GPT3.5, GPT4o, ORCA-2, or LLaMA-2.

[0030] During inference, prompts 22 are received via prompt word interface 24. In one set of examples, prompt word interface 24 may be a graphical user interface of a program such as a chatbot, browser, or productivity application; in another example, prompt word interface 24 may be an application programming interface. Prompts 22 consist of text data, which may include unstructured text such as natural language input, and may also include data that can be processed by the preprocessor 26 or postprocessor 27 in model plugin 20 (see [link to model plugin]). Figure 2 The structured text is interpreted as follows. The cue word 22 includes a first cue word portion 28 and a second cue word portion 30. When using a multimodal model, the cue word 22 may include other input modalities, such as images or audio.

[0031] The first prompt word portion 28 includes associated source metadata 29, which indicates the data source 44 used to retrieve the data source 52. For example, the first prompt word portion 28 could be in the form of structured text that defines how the data retrieved from the data source 44 should be presented in the output sequence 38. As a concrete example, the first prompt word portion 28 could be a manifest encoded in JavaScript Object Notation (JSON) that defines keys and values, and the source metadata 29 could include a link to the data source 44 to populate the values ​​associated with the keys defined in the JSON manifest. As yet another example, discussed in more detail below, the data source 44 could reside in a database 60 (see...). Figure 5 (discussed below), and the first source metadata 29 may include the location of the evidenced data source 44 in the database 60. The processing circuitry 12 is configured to obtain the first output portion 38A at least in part by performing a database lookup operation at the database 60. As another example, the first portion 28 may include natural language prompt words, and the evidenced data source 44 may be a language model that has been fine-tuned using domain-specific knowledge and equipped with post-generation verification logic to improve its accuracy. In yet another example, the first source metadata includes author information associated with a verified user account. In this way, the source metadata can identify the original author of the first prompt word portion 28 of the prompt word 22.

[0032] The second prompt word section 30 may include one or more instructions for the machine learning model 18, as well as contextual data related to how the prompt word should be answered, such as the expected author, audience, style, length, and language of the desired response. Background material or source document fragments for context may also be included in the second prompt word section 30.

[0033] The prompt word 22 is passed through the lexicon 32, which lexiconizes the text and other data in the prompt word 22 to produce a lexiconized prompt word 33, which includes an input sequence 34 consisting of input lexicons 36. The preprocessor 26 of the model plugin 20 is configured to receive the lexiconized prompt word 33. It should be understood that the lexiconized prompt word 33 includes a first prompt word portion 33A and a second prompt word portion 33B. The first prompt word portion 33A includes one or more first input lexicons 33A1, which are marked with first source metadata lexicons 33A2 indicating a data source 44. The second prompt word portion 33B includes one or more second input lexicons 33B1 that do not have first source metadata.

[0034] The data source 44 is typically accessed by the data content module 42 via a network 46, such as the Internet or a local area network. The preprocessor 26 is configured to parse the prompt word 22 and create a parse tree of the content contained therein. For example, preprocessor instructions can be inserted into the prompt word 22 to identify the first prompt word portion 28, the second prompt word portion 30, and the source metadata 29 before processing by the preprocessor 26, thereby enabling the preprocessor 26 to create the parse tree. After parsing the prompt word 22, the preprocessor 26 is configured at 26A to determine whether the prompt word contains data content 52 referenced by the first prompt word portion 28 (encoded in the first input lexicon 33A) and the associated source metadata 29 (encoded in the source metadata lexicon 33A2). Upon a positive determination, the preprocessor 26 is configured to invoke the associated data content module 42 of the model plugin 20. A link (e.g., a URL or URI) to the data source 44 is passed to the data content module 42. The link may include a network address, a path, and one or more parameters or status identifiers extracted from the source metadata 29 contained in the lexicalized first source metadata 33A2 (e.g., the status identifier may be obfuscated in a GUID). Such a link may be referred to as a deep link. The data content module 42 uses the link to retrieve data content from the data source 44 via the computer network 46. Therefore, the processing circuit 12 can be configured to receive a parse tree that specifies the corresponding positions of the first output portion 38A and the second output portion 38B in the output sequence 38. After determining that sufficient information has been obtained to continue generating the output sequence by the machine learning model ("yes" at 26B), the model plugin 20 is configured to pass the lexicalized cue word 33 with the input sequence 34 to the machine learning model 18 and generate the output sequence 38 as specified by the parse tree.

[0035] The processing circuit 12 is also configured to generate the output sequence 38 of output tokens 40 at least in part by obtaining a first output portion 38A of the output sequence 38 from the documented data source 44 indicated in the first source metadata 33A2. The processing circuit 12 is also configured to generate a second output portion 38B of the output sequence 38 at the machine learning model 18 based at least in part on the second cue word portion 33B and the retrieved first output portion 38A. The first output portion 33A contains the first output token 40A, and the second output portion 33B contains the second output token 40B. It should be understood that the number of input tokens and output tokens is shown in a simplified form; therefore, where one token is shown, multiple tokens can be represented.

[0036] This generation, performed by machine learning model 18, is carried out using transformer 18A in an autoregressive word-by-word generation loop 39 until the output sequence 38 is completed. Each time the generation loop passes, transformer 18A generates a probability distribution of candidate words for the next output word in the output sequence 38. One word from the candidate words is sampled according to a sampling function that can take one or more sampling parameters (e.g., a temperature parameter) as input to adjust the sampling method. This process continues along generation loop 39 until machine learning model 18 completes the generation of output sequence 38. In this way, by implementing machine learning model 18, processing circuitry 12 is configured to compute the second output portion 38B, at least in part, based on the lexicalized cue word 33 and the probabilistically generated words selected so far in the second output portion 38B of output sequence 38, via autoregressive generation at each stage of word-by-word generation loop 39.

[0037] Once output sequence 38 is complete, machine learning model 18 is configured to transfer output sequence 38 to additional computational processes, such as file storage, transmission, display, or downstream application processes. As shown, output sequence 38 can be de-lexicalized by a lexer, and response 48 can be generated. Response 48 includes text data comprising: model-generated content 50 based on the second output portion 38B of output sequence 38; data-driven content 52 based on the first output portion 38A of output sequence 38; and a link 54 to data-driven data source 44, which may also be encoded in the first output portion 38A.

[0038] Response 48 can be output to prompt interface 24. Therefore, in one example, the aforementioned additional calculation process can be a graphical user interface (GUI), and processing circuitry 12 can be configured to transmit output sequence 38 for display at the GUI, where model-generated content 50 encoded in the second output section 38B and edified content 52 encoded in the first output section 38A are visually distinguishable. Text associated with the first output section 38A can be marked at the GUI with an indicator of edified data source 44 (e.g., a link to edified data source 44). Examples are discussed below. Figure 2 As shown, the model-generated content uses plain text, while the data-driven content uses bold text. Of course, other forms of emphasis can also be used, such as color, size, font, outline, underline, highlight, and quotations. In another example, the prompt word interface can be an API or a storage interface, and the additional computation process can be a downstream computation program, such as a prompt word API or storage interface.

[0039] The computing system 10A can be configured to insert non-training labels 31 into the model output using the evidenced content lexical insertion technique described herein, thereby preventing or prohibiting the training of third-party models based on the output of machine learning model 18 and / or evidenced data source 44. To this end, processing circuitry 12 can be configured to: label the first output portion 38A with source metadata 29 as first source metadata, and / or label the second output portion 38B with second source metadata 29A indicating the output generated by the machine learning model. Processing circuitry 12 can also be configured to exclude output sequence 38 from the training corpus of the additional machine learning model, at least in part, based on determining that the output sequence is labeled with the first source metadata 29 or the second source metadata 29A. This can be achieved by inserting non-training labels 31 associated with the first output portion 38A and / or the second output portion 38B into the training data as appropriate. In this way, the output of machine learning model 18 and / or the output of evidenced data source 44 can be avoided when training the additional machine learning model.

[0040] Now go to Figure 4 This describes an example use case scenario for a computing system 10A that utilizes evidenced lexical insertion. As shown in the figure, a prompt word is received, the first part 28 of which includes a citation to a leading work in computer science entitled "The Authoritative and Latest Guide to Innovation in Computer Science." Prompt word 22 includes a textual citation of the authoritative work and a link to the work available online to the user, as well as source metadata 29 indicating the authoritative work's web location at evidenced data source 44. In the second part 30 of prompt word 22, instructions are provided to generate a graduation speech for computer science graduates at a university graduation ceremony, specifically citing three advancements in computer science technology that have occurred in the past four years as described in the cited evidenced content source.

[0041] The prompt word 22 is lexicalized to generate an input sequence 34, which is passed to model plugin 20. Model plugin 20 executes preprocessor 26, which then parses the lexicalized prompt word 22 and determines the existence of evidenced content with source metadata 29, which indicates the location of an evidenced data source 44 containing the authoritative works cited in the prompt word ("Yes" at 26A). Preprocessor 26 invokes evidenced content module 42, which then retrieves information for "three technological breakthroughs in the references". Since this is a natural language description of the information requested from evidenced data source 44, the information can be retrieved using the language model interface at evidenced data source 44. Alternatively, a vector database comparison with similarity search and ranking techniques can be used to match the query with paragraphs in evidenced data source 44. In this illustrated example, the following texts were retrieved: (1) “GPT-3 has over 175 billion parameters, and GPT-4o is believed to be even larger,” (2) “StableDiffusion3 can generate photorealistic images and includes models with up to 8 billion parameters that run via a backdiffusion process,” and (3) “Quantum computers with over 1000 qubits have been built,” along with links associated with each item. Following the retrieval, the lexicalized text for these sentences was inserted as evidenced content lexical 40A into the output sequence 38, interleaved with model-generated lexical 40B.

[0042] To insert the edible content lexicon 40A into the model-generated lexicon 40B during the autoregressive word-by-word generation loop 39, the input prompt word is passed to the machine learning model 18 to begin generation, and a parse tree for the output is generated and followed. This parse tree contains templates for the edible content insertion points. In this way, the edible content is written directly into the output sequence 38, avoiding sending the edible content through the machine learning model 18, thereby improving model efficiency (attributed to reduced computation) and accuracy (attributed to the absence of edible content in the probabilistic model generation process). Figure 4 The full text of the graduation speech generated by the computing system 10A in response 48 is shown, including interleaved model-generated content 50 and evidence-based content 52.

[0043] Figure 5Another example configuration of a computing system 10A is shown, which includes a post-processor 27 for a model plugin 20. In this configuration, a prompt 22 is prepared, comprising a first part 28 that includes a reference to a database 60. The database reference includes structured text with an SQL query for retrieving data from a specific column and row of a particular table, and a link to that table. The source metadata 29 in the prompt includes a network address at which the database 60, used as the source of the data content 44, is located. The prompt 22 also includes a second part 30 that includes instructions to prepare a paragraph about a company for an annual report, summarizing the company's products and office locations, and including the total number of company employees retrieved from the referenced database 60 link. The prompt 22 is tokenized and passed to the model plugin 20.

[0044] Preprocessor 26 processes the prompt words to generate a parse tree and identifies the data content in the first part 28 and the instructions in the second part 30. After determining that the data content is referenced in the prompt word 22, preprocessor 26 calls data content module 42 to insert a data content insertion indicator, such as insertion point lexical 40C, at the position where the data content will be inserted later in the output sequence 38. Then, the input sequence 34 of the lexicalized prompt words is passed to machine learning model 18, which performs word-by-word generation of the output sequence 38 in an autoregressive word-by-word generation loop 39. Subsequently, machine learning model 18 generates output lexical 40 in the output sequence. After the entire output sequence 38 has been generated (or alternatively, after the output sequence has been generated up to insertion point lexical 40C), postprocessor 27 of model 18 calls data content module 42 to obtain the data content portion from data data source 44, in this case, which involves a database call to database 60 to select data at the row and column indicated in source data 29. In this embodiment, a database lookup operation is performed at database 60 to retrieve data-rich content and return it to data-rich content module 42. In some embodiments, the data-rich data source 44 can be indicated in the data-rich content insertion indicator, and the insertion point lexicon 40C can spell out the position of the data-rich data source 44. In such an embodiment, the insertion point lexicon 40C is replaced, and the output sequence is updated by replacing the data-rich content insertion indicator with the data-rich output portion. The updated output sequence 38U is converted to text via lexiconizer 32 through delexiconization and is transmitted to additional computational processes, such as display, storage, or APIs of downstream applications. As shown, the generated updated output sequence 38U is delexiconized by the lexiconizer to become text and is output as response 48. The depicted response 48 includes model-generated content 50, which takes the form of a company description requested by the instruction in prompt 22; it also includes the total number of employees, i.e., 450, retrieved from database 60 as data-rich content 52. The response also includes a link to the data source 44 from which the data content 52 is retrieved.

[0045] Figure 6A flowchart of method 80 according to a first embodiment of the present disclosure is shown, the method for detecting source metadata in an output sequence of a machine learning model, retrieving credentialed content using the source metadata, and inserting the credentialed content into the output sequence. At 82, the method includes receiving a lexicalization cue word at the machine learning model. At 84, the method includes a model-generated content portion of the output sequence that generates output lexicals in response to the lexicalization cue word. At 86, the method includes identifying source metadata for a credentialed data source in the model-generated content portion of the output sequence. At 88, the method includes at least temporarily halting the probabilistic generation of the output sequence using the machine learning model. At 90, the method includes retrieving credentialed content from the credentialed data source using the source metadata. At 92, the method includes writing output lexicals corresponding to the credentialed content into the credentialed content portion of the output sequence. At 94, the method includes transferring the output sequence to an additional computation process. As shown at 96, the additional computation process may be a graphical user interface (GUI), and the method may further include transferring the output sequence for display at the GUI, wherein the model-generated content portion and the credentialed content portion are visually distinguishable. In addition, data-related content sections can be marked with indicators of the data source in the GUI.

[0046] Method 80 may further include: identifying the edified content portion using source metadata as first source metadata, and identifying the model-generated content portion using second source metadata that instructs the machine learning model to generate the output. Furthermore, the output sequence may be excluded from the training corpus of the additional machine learning model, at least in part, based on the determination that the output sequence is labeled with second source metadata.

[0047] The model-generated content portion can be a first model-generated content portion, and the method may further include: at the machine learning model, computing a second model-generated output portion via autoregressive generation based at least in part on the context including lexicalized cue words, the edified content portion of the output sequence, and the first model-generated output portion. In some examples, the first source metadata includes author information associated with a verified user account. In other examples, the first source metadata includes the location of the edified data source in a database, and the method further includes obtaining the edified content portion at least in part by performing a database lookup operation at the database.

[0048] Figure 7AA flowchart of a method 100 for use with a computing system to insert data-driven content into a response generated at a machine learning model is shown. Method 100 can be implemented using the aforementioned computer hardware and software components or other suitable computer hardware and software. At step 102, method 100 includes receiving a lexicalized prompt. For example, the lexicalized prompt can be computed at a lexer based on natural language input received at a GUI. The lexicalized prompt includes a first prompt portion and a second prompt portion. The first prompt portion includes one or more first input lexical units labeled with first source metadata indicating a data source. The second prompt portion includes one or more second input lexical units that do not have the first source metadata.

[0049] In some examples, the first-source metadata may include the location of the credentialed data source in a database. As another example, the first-source metadata may include author information associated with a verified user account. As yet another example, the first-source metadata may include a hyperlink to the credentialed data source. Therefore, the first-source metadata may be a note instructing a machine learning model to generate content from a credentialed data source outside of the content.

[0050] At step 104, method 200 further includes generating an output sequence at least partially based on lexicalized cue words. Generating the output sequence at step 104 includes: at step 106, obtaining a first output portion of the output sequence from a credentialed data source indicated by the first source metadata. In an example where the first source metadata indicates a location in a database, obtaining the first output portion at step 106 may include: at step 108, performing a database lookup operation at the database. In other examples, some other data structure or procedure may be used as the credentialed data source.

[0051] At step 110, generating the output sequence at step 104 further includes generating a second output portion of the output sequence at a machine learning model. The machine learning model can be a Large Language Model (LLM) or a Multimodal Model (LMM). The second output portion is generated at least in part based on a second cue word portion and the retrieved first output portion. For example, at step 112, step 110 may include computing the second output portion at least in part based on context via autoregressive generation. In such an example, the context includes a lexicalized cue word, a first portion of the output sequence, and a previous output sequence included in the second output portion. The previous output sequence is initialized to an empty set in the first autoregressive generation iteration and constructed by iteratively adding second output lexicals generated as part of the second output portion in subsequent autoregressive generation iterations. In the example of performing step 112, including evidenced content in the context can reduce illusions during the computation of the second output portion.

[0052] At step 114, method 100 further includes transferring the output sequence to an additional computation process. For example, the additional computation process could be a GUI (User-Defined Interface). In such an example, step 114 may also include: at step 116, transferring the output sequence for display at the GUI, wherein the first output portion and the second output portion are visually distinguishable. Additionally, the first output portion can be marked at the GUI with an indicator of the credible data source. Therefore, the user can easily identify credible content and machine learning model-generated content within the output sequence. In an example where the first source metadata includes a hyperlink, the hyperlink can be provided to the user at the GUI as an indicator of the credible data source. Therefore, the user can quickly and easily refer to the credible data source to verify credible content or obtain further information.

[0053] Figures 7B-7C Additional steps of method 100 that can be performed in some examples are shown. At step 118, as... Figure 7B As shown, method 100 may further include receiving a parse tree that specifies the corresponding positions of the first and second output portions in the output sequence. At step 120, method 100 may further include generating the output sequence as specified by the parse tree. Therefore, a more complex data content insertion structure than a single data content insertion position can be specified in the lexical prompt. For example, nested references to data content can be included in the output sequence.

[0054] Figure 7C Additional steps 122 and 124 that can be performed when generating the output sequence are shown, as well as step 126 that can be performed after the output sequence is output. At step 122, method 100 may further include identifying a first output portion with first source metadata. Additionally, at step 124, method 100 may further include identifying a second output portion with second source metadata instructing the machine learning model to generate the output.

[0055] After outputting this output sequence, according to Figure 7C For example, the first and second output portions can be processed differently based on their metadata. At step 126, method 100 may further include excluding the output sequence from the training corpus of the supplementary machine learning model, at least in part, based on determining that the output sequence is tagged with second source metadata. Therefore, the supplementary machine learning model can be trained in a way that avoids training on outputs generated by the machine learning model that may include hallucinations.

[0056] Figure 8 A flowchart of method 200 is shown, which can be performed in some examples as another method for inserting evidenced content into the output of a machine learning model. At step 202, method 200 includes receiving lexicalized cue words.

[0057] At step 204, method 200 further includes generating an output sequence at the machine learning model, at least in part, based on lexicalized cue words. The output sequence includes a data-driven content insertion indicator, which may be, for example, a single output lexical or a sequence of multiple output lexicals. The data-driven content insertion indicator specifies a data-driven data source and acts as a placeholder for data-driven content. In some examples, the output sequence includes a parse tree. In such examples, the parse tree may specify a structure in which data-driven content and machine learning model-generated content are arranged within the output sequence.

[0058] At step 206, method 200 further includes obtaining a data-driven output portion from a data-driven data source indicated by a data-driven content insertion indicator. For example, in some examples, the data-driven content insertion indicator may include the location of the data-driven data source in a database. In such an example, step 206 may include obtaining the data-driven output portion, at least in part, by performing a database lookup operation at the database at step 208. As another example, the data-driven content insertion indicator may include author information associated with a verified user account. In such an example, data-driven content received from a verified user account may be obtained at step 206.

[0059] At step 210, method 200 further includes updating the output sequence at least in part by replacing the data content insertion indicator with a data-based output portion. The data-based output portion may also be tagged with source metadata specifying a data-based data source.

[0060] At step 212, method 200 further includes transferring the updated output sequence to an additional computation process. In some examples, the additional computation process may be a GUI. In such examples, method 200 may also include, at step 214, transferring the updated output sequence for display at the GUI, wherein the credible output portion and the machine learning model-generated portion of the updated output sequence are visually distinguishable. Thus, credible content can be indicated in a way that is quickly and easily identifiable by the user. The credible output portion may also be annotated with indications of the credible data source.

[0061] The aforementioned system and method can be used to include references to credible content and credible data sources, as well as instructions for generating content via a machine learning model, in prompt word 22. This allows the machine learning model to retrieve credible content from the credible content source and directly write the credible content into the appropriate position in the model's output sequence, interleaving it with model-generated content. The writing of output terms can be performed by the preprocessor 26 or postprocessor 26 of the model plugin as described above. Directly retrieving credible content and writing it into the output sequence in this manner achieves the technical effect of improving the efficiency of the machine learning model, because the credible content does not need to pass through the machine learning model during probabilistic inference, thereby reducing the computational resources, energy, and time used for inference; and by directly passing the credible content to the output sequence in an unmodified form without subjecting it to probabilistic generation, the accuracy of the model is also improved. Furthermore, including source metadata in the generated response improves the credibility and traceability of both model-generated content and credible content. In addition, by providing credible content sources with up-to-date or proprietary information sources, the content of the generative model can be enhanced in an efficient and accurate manner by utilizing updated, more valuable, or more significant information obtained from these credible content sources.

[0062] In some embodiments, the methods and processes described herein can be attached to a computing system comprising one or more computing devices. Specifically, such methods and processes can be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0063] The methods and processes described herein are tied to a computing system consisting of one or more computing devices. Specifically, such methods and processes can be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0064] Figure 9 A non-limiting embodiment of a computing system 300 is illustrated schematically, which may implement one or more of the methods and processes described above. The computing system 300 is shown in a simplified form. The computing system 300 may embody the computing systems 10 and 10A described above. Components of the computing system 300 may be included in one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, video game devices, mobile computing devices, mobile communication devices (e.g., smartphones), and / or other computing devices, and may also be included in wearable computing devices such as smartwatches and head-mounted augmented reality devices.

[0065] The computing system 300 includes processing circuitry 302, volatile memory 304, and non-volatile storage device 306. The computing system 300 may optionally include a display subsystem 308, an input subsystem 310, a communication subsystem 312, and / or... Figure 9Other components not shown.

[0066] Processing circuitry 302 typically includes one or more logic processors, which are physical devices configured to execute instructions. For example, a logic processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, achieve technical effects, or otherwise obtain desired results.

[0067] The logic processor may include one or more physical processors configured to execute software instructions. Additionally or alternatively, the logic processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of processing circuit 302 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential processing, parallel processing, and / or distributed processing. Components of processing circuit 302 may optionally be distributed across two or more separate devices, which may be located remotely and / or configured for coordinated processing. For example, aspects of the computing system 300 disclosed herein may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration. In this case, it should be understood that these virtualized aspects operate on different physical logic processors on various different machines. It should be understood that these different physical logic processors on different machines are collectively included in processing circuit 302.

[0068] The non-volatile storage device 306 includes one or more physical devices configured to store instructions executable by processing circuitry to implement the methods and processes described herein. When such methods and processes are implemented, the state of the non-volatile storage device 306 can be transformed, for example, to store different data.

[0069] Non-volatile storage device 306 may include removable and / or built-in physical devices. Non-volatile storage device 306 may include optical memory, semiconductor memory, and / or magnetic memory, or other high-capacity storage device technologies. Non-volatile storage device 306 may include non-volatile, dynamic, static, read / write, read-only, sequential access, location-addressable, file-addressable, and / or content-addressable devices. It should be understood that even if non-volatile storage device 306 is powered off, non-volatile storage device 306 is configured to retain instructions.

[0070] Volatile memory 304 may include a physical device containing random access memory. Volatile memory 304 is typically used by processing circuitry 302 to temporarily store information during the processing of software instructions. It should be understood that when volatile memory 304 is powered off, it typically will not continue storing instructions.

[0071] Various aspects of the processing circuitry 302, the volatile memory 304, and the non-volatile storage device 306 can be integrated together into one or more hardware logic components. For example, such hardware logic components may include field-programmable gate arrays (FPGAs), application-specific integrated circuits (PASICs / ASICs), application-specific standard products (PSSPs / ASSPs), system-on-a-chip (SOCs), and complex programmable logic devices (CPLDs).

[0072] The terms "module," "program," and "engine" can be used to describe an aspect of computing system 300, typically implemented in software by a processor, that uses a portion of volatile memory to perform a specific function involving transformation processing of the processor specifically configured to perform that function. Therefore, a module, program, or engine can be instantiated by processing circuitry 302 executing instructions held by non-volatile memory 306 and using a portion of volatile memory 304. It should be understood that different modules, programs, and / or engines can be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine can be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" can encompass single or grouped executable files, data files, libraries, drives, scripts, database records, etc.

[0073] When included, the display subsystem 308 can be used to present a visual representation of the data stored by the non-volatile storage device 306. This visual representation may take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data stored by the non-volatile storage device 306, and thus change the state of the non-volatile storage device 306, the state of the display subsystem 308 can also be changed to visually represent changes in the underlying data. The display subsystem 308 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with the processing circuitry 302, the volatile memory 304, and / or the non-volatile storage device 306 in a shared housing, or such display devices may be peripheral display devices.

[0074] When included, the input subsystem 310 may include one or more user input devices (such as a keyboard, mouse, touchscreen, camera, or microphone) or interface with such devices.

[0075] When included, the communication subsystem 312 can be configured to communicatively couple the various computing devices described herein to each other and to other devices. The communication subsystem 312 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem 312 may be configured to communicate via wired or wireless local area networks or wide area networks, broadband cellular networks, etc. In some embodiments, the communication subsystem 312 may allow the computing system 300 to send messages to and / or receive messages from other devices via a network such as the Internet.

[0076] The following paragraphs discuss several aspects of this disclosure. According to one aspect of this disclosure, a computing system is provided, including processing circuitry and associated memory storing instructions that, when executed, cause the processing circuitry to: receive a lexicalized cue word; generate a model-generated content portion of an output sequence of output lexicalized words in response to the lexicalized cue word; identify source metadata for a data source in the model-generated content portion of the output sequence; at least temporarily halt the probabilistic generation of the output sequence per lexicalized word using a machine learning model; retrieve data-generated content from the data source using the source metadata; write the output lexicalized words corresponding to the data-generated content into the data-generated content portion of the output sequence; and transmit the output sequence to an additional computing process.

[0077] In this respect, the additional computation process can be a graphical user interface (GUI); and the processing circuitry can be configured to transmit an output sequence for display at the GUI, wherein the model-generated content of the model-generated content portion based on the output sequence and the edible content portion based on the output sequence are visually distinguishable. Furthermore, in this respect, the edible content can be marked at the GUI with an indicator of the edible data source.

[0078] In this respect, the source metadata can be first source metadata, and the processing circuit can also be configured to: identify the data-generated content portion with the first source metadata; and identify the model-generated content portion with second source metadata indicating the output generated by the machine learning model. Furthermore, in this respect, the processing circuit can also be configured to exclude the output sequence from the training corpus of the additional machine learning model, at least in part, based on the determination that the output sequence is identified with the second source metadata.

[0079] In this respect, the model-generated content portion can be a first model-generated content portion, and at the machine learning model, the processing circuitry can also be configured to compute a second model-generated output portion via autoregressive generation, at least in part based on the context of the evidenced content portion including lexicalized cue words, the output sequence, and the first model-generated output portion.

[0080] In this regard, source metadata may include author information associated with a verified user account.

[0081] In this respect, the source metadata may include the location of the data source in the database; and the processing circuit may also be configured to obtain the data-driven output portion at least in part by performing a database lookup operation at the database.

[0082] In this respect, the processing circuit can also be configured to: receive a parse tree specifying the corresponding positions of the first output portion and the second output portion in the output sequence; and generate the output sequence as specified by the parse tree.

[0083] According to another aspect, a computing system is provided, including processing circuitry and associated memory storing instructions that, when executed, cause the processing circuitry to: receive lexicalized cue words; at a machine learning model, generate an output sequence at least in part based on the lexicalized cue words, wherein the output sequence includes a data-driven content insertion indicator; obtain a data-driven output portion from a data-driven data source indicated by the data-driven content insertion indicator; update the output sequence at least in part by replacing the data-driven content insertion indicator with the data-driven output portion; and transmit the updated output sequence to an additional computing process.

[0084] In this respect, the additional computation process can be a graphical user interface (GUI); and the processing circuitry can also be configured to transmit an updated output sequence for display at the GUI, wherein the data-based content of the data-based output portion and the model-generated content of the machine learning model-generated portion of the updated output sequence are visually distinguishable.

[0085] In this regard, funded output can be marked with an indicator of the funded data source in the GUI.

[0086] In this regard, evidenced content insertion indicators may include author information associated with a verified user account.

[0087] In this respect, the data-driven content insertion indicator may include the location of the data-driven data source in the database; and the processing circuitry is also configured to obtain the data-driven output portion at least in part by performing a database lookup operation at the database.

[0088] According to another aspect, a method for use with a computing system is provided, the method comprising: receiving a lexicalized cue word; generating a model-generated content portion of an output sequence of output lexicals in response to the lexicalized cue word; identifying source metadata for a data source in the model-generated content portion of the output sequence; at least temporarily suspending probabilistic generation of the output sequence per lexical using a machine learning model; retrieving data-generated content from the data source using the source metadata; writing the output lexicals corresponding to the data-generated content into the data-generated content portion of the output sequence; and transmitting the output sequence to an additional computing process.

[0089] In this respect, the additional computation process may be a graphical user interface (GUI); and the method may also include transmitting an output sequence for display at the GUI, wherein model-generated content based on the model-generated content portion and data-generated content based on the data-generated content portion are indicated in a visually distinguishable manner; and the data-generated content may be marked at the GUI with an indicator of the data-generated data source.

[0090] In this respect, the source metadata may be first source metadata, and the method may further include: identifying the edified content portion with the first source metadata; identifying the model-generated content portion with second source metadata that instructs the machine learning model to generate the output; and excluding the output sequence from the training corpus of the additional machine learning model, at least in part based on the determination that the output sequence is identified with the second source metadata.

[0091] In this respect, the model-generated content portion may be a first model-generated content portion, and the method may further include: at the machine learning model, at least in part based on the context of the evidenced content portion including lexicalized cue words, the output sequence, and the first model-generated output portion, computing a second model-generated output portion via autoregressive generation.

[0092] In this regard, the first-source metadata may include author information associated with a verified user account.

[0093] In this respect, the first source metadata may include the location of the data source in the database; and the method may also include obtaining the data content portion at least in part by performing a database lookup operation at the database.

[0094] As used in this article, “and / or” is defined as inclusive OR ∨, as shown in the following truth table:

[0095] It should be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered limiting, as many variations may exist. The particular routines or methods described herein may represent one or more of any number of processing strategies. Therefore, the various actions illustrated and / or described may be performed in the illustrated and / or described order, in a different order, in parallel, or omitted. Similarly, the order of the above processes may be changed.

[0096] The subject matter of this disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations disclosed herein, as well as any and all equivalents thereof.

Claims

1. A computing system (10), comprising: The processing circuit (12) and its associated memory (14) store instructions (16) that, when executed, cause the processing circuit to: Receive word meta-prompt (33); In response to the lexicalized prompt, a model-generated content part (40B) of the output sequence (38) of the output lexical (40) is generated; Source metadata (29) for the data source (44) in the model-generated content portion that identifies the output sequence; At least temporarily suspend the word-by-word probabilistic generation of the output sequence using the machine learning model (18); Retrieve datatard content from the datatard data source using the source metadata (52); Write the output tokens corresponding to the datated content into the datated content portion (40A1) of the output sequence; and The output sequence is then transmitted to an additional computational process.

2. The computing system according to claim 1, wherein: The additional calculation process is a graphical user interface (GUI); and The processing circuitry is configured to transmit the output sequence for display at the GUI, wherein the model-generated content of the model-generated content portion based on the output sequence and the data-generated content of the data-generated content portion based on the output sequence are indicated in a visually distinguishable manner.

3. The computing system according to claim 2, wherein, The data content is marked with an indicator of the data source in the GUI.

4. The computing system according to any one of claims 1-3, wherein, The source metadata is first source metadata, and the processing circuit is further configured to: The documented content portion is identified using the first source metadata. as well as The model-generated content portion is identified by a second source metadata indicating the output generated by the machine learning model.

5. The computing system according to claim 4, wherein, The processing circuitry is also configured to exclude the output sequence from the training corpus of the additional machine learning model, at least in part based on determining that the output sequence is tagged with the second source metadata.

6. The computing system according to any one of claims 1-5, wherein, The model-generated content portion is a first model-generated content portion, and wherein, at the machine learning model, the processing circuitry is further configured to compute a second model-generated output portion via autoregressive generation, at least in part, based on a context including the following: The word meta-catalyst; The datated content portion of the output sequence; and The first model generates the output portion.

7. The computing system according to any one of claims 1-6, wherein, The source metadata includes author information associated with verified user accounts.

8. The computing system according to any one of claims 1-7, wherein: The source metadata includes the location of the data source in the database; and The processing circuitry is also configured to obtain the data-rich output portion at least in part by performing a database lookup operation at the database.

9. The computing system according to any one of claims 1-8, wherein, The processing circuit is further configured to: Receive a parse tree, the parse tree specifying the corresponding positions of the first output portion and the second output portion in the output sequence; and Generate the output sequence as specified in the parse tree.

10. A computing system (10), comprising: The processing circuit (12) and its associated memory (14) store instructions (16) that, when executed, cause the processing circuit to: Receive word meta-prompt (33); At the machine learning model (18), an output sequence (38) is generated at least in part based on the lexicalized cue words, wherein the output sequence includes a data-based content insertion indicator; The data output portion (40A1) is obtained from the data source (44) indicated by the data content insertion indicator; The output sequence is updated at least in part by replacing the datated content insertion indicator with the datated output portion; and The updated output sequence is then transferred to the additional computation process.

11. The computing system according to claim 10, wherein: The additional calculation process is a graphical user interface (GUI); and The processing circuitry is also configured to transmit the updated output sequence for display at the GUI, wherein the data-based content of the updated output sequence based on the data-based output portion and the model-generated content based on the machine learning model-generated portion are visually distinguishable.

12. The computing system according to claim 11, wherein, The data output is marked with an indicator of the data source in the GUI.

13. The computing system according to any one of claims 10-12, wherein, The evidenced content insertion indicator includes author information associated with a verified user account.

14. The computing system according to any one of claims 10-13, wherein: The data-driven content insertion indicator includes the location of the data source in the database; and The processing circuitry is also configured to obtain the data-rich output portion at least in part by performing a database lookup operation at the database.

15. A method (80) for use with a computing system, the method comprising: Receive (82) word dichotomy prompts; In response to the lexicalized prompt, a model-generated content portion of the output sequence of output lexical units is generated (84); Identifier (86) is the source metadata for the data source in the model-generated content portion of the output sequence; At least temporarily suspend (88) the word-by-word probabilistic generation of the output sequence using the machine learning model; Retrieve (90) documented content from the documented data source using the source metadata; Write the output tokens corresponding to the datatended content into the datatended content portion of the output sequence (92); and The output sequence is transmitted (94) to the additional computation process.

16. The method of claim 15, wherein: The additional calculation process is a graphical user interface (GUI); The method further includes transmitting the output sequence for display at the GUI, wherein the model-generated content based on the model-generated content portion and the data-generated content based on the data-generated content portion are visually distinguishable; and The data content is marked with an indicator of the data source in the GUI.

17. The method according to claim 15 or 16, wherein, The source metadata is first source metadata, and the method further includes: The documented content portion is identified using the first source metadata; and The model-generated content portion is identified using second-source metadata indicating the output of the machine learning model; and The output sequence is excluded from the training corpus of the additional machine learning model, at least in part, based on the determination that the output sequence is tagged with the second source metadata.

18. The method according to any one of claims 15-17, wherein, The model-generated content portion is a first model-generated content portion, and the method further includes: at the machine learning model, at least in part based on a context including the following, a second model-generated output portion is computed via autoregressive generation: The word meta-catalyst; The datated content portion of the output sequence; and The first model generates the output portion.

19. The method according to any one of claims 15-18, wherein, The first source metadata includes author information associated with a verified user account.

20. The method according to any one of claims 15-19, wherein: The first source metadata includes the location of the data source in the database; and The method also includes obtaining the data-rich portion at least in part by performing a database lookup operation at the database.