Medical query processing with sequence processing models

Abstract

Provided are a number of different systems and methods that enhance the performance and reliability of sequence processing models, particularly when applied to the processing of medical data and queries. The proposed techniques collectively address the challenges of integrating vast, evolving external knowledge sources, refining responses in the face of uncertainty, and efficiently managing complex, multi-modal datasets such as are commonly found in the medical field. Specifically, by leveraging iterative self-training, uncertainty-guided retrieval, multi-stage prompting, and / or advanced model architectures, the disclosed technology improves how machine learning models can process, analyze, and / or utilize complex information, resulting in improved computer systems applicable to a number of different applications or use cases, including medical diagnostics and / or other medical applications.

Description

MEDICAL QUERY PROCESSING WITH SEQUENCE PROCESSING MODELSPRIORITY CLAIM

[0001] The present application is based on and claims priority to United States Provisional Application Number 63 / 639,476 having a filing date of April 26, 2024. Application claims priority to and the benefit of each of such applications and incorporates all such applications herein by reference in their entirety.FIELD

[0002] The present disclosure relates generally to machine learning processes and machine-learned devices and systems. More particularly, the present disclosure relates to systems and methods improving the ability of sequence processing models to leverage external sources of knowledge and process large volumes of context data, for example in furtherance of processing medical queries.BACKGROUND

[0003] The application of sequence processing models, such as large language models (LLMs) and large multimodal models (LMMs), in complex domains poses numerous technical challenges. In healthcare, for instance, conventional models often struggle to provide accurate and reliable answers to medical queries that require complex clinical reasoning or knowledge of the latest medical research. These models typically fall short in integrating real-time data from external sources, which can assist with providing responses consistent with evolving medical knowledge. Additionally, there is a need for optimization in the models’ learning processes to better simulate complex decision-making required in clinical settings.

[0004] Moreover, medical professionals often grapple with the task of sifting through large and complex datasets, such as extensive electronic health records (EHRs), high- resolution medical images, and genomic sequences. Conventional data processing systems may become overwhelmed by the volume and variety of this data, leading to inefficiencies and difficulties in extracting precise and relevant information.

[0005] Thus, the technical problems faced by healthcare information technology systems include ensuring the generation of reliable and accurate medical responses from Alsystems amidst complex medical data and doing so efficiently to minimize resource usage and expedite the decision-making process.SUMMARY

[0006] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

[0007] One example aspect of the present disclosure is directed to a computer- implemented method to perform self-training of sequence processing models to leverage external sources of knowledge. The method includes performing, by a computing system comprising one or more computing devices, a plurality of self-training iterations. Each selftraining iteration comprises: obtaining, by the computing system, an exemplar tuple comprising an exemplar query and an exemplar response, wherein the exemplar response comprises an exemplar chain of analysis that arrives at an exemplar answer to the exemplar query, and wherein the exemplar chain of analysis relies upon or makes reference to one or more exemplar items of content obtained from one or more external sources of knowledge; generating, by the computing system, a training prompt comprising the exemplar tuple, a training query, and one or more training items of content obtained from the one or more external sources of knowledge; processing, by the computing system, the training prompt with a sequence processing model to generate a candidate response as an output of the sequence processing model, wherein the candidate response comprises a model -generated chain of analysis that arrives at a candidate answer to the training query; determining, by the computing system, whether the candidate response corresponds to a ground truth answer to the training query; and in response to determining that the candidate response corresponds to the ground truth answer, fine-tuning the sequence processing model on the candidate response. Other example aspects are directed to computer systems and non-transitory computer-readable media that are configured to perform or enable performance of the method.

[0008] Another example aspect of the present disclosure is directed to a computer- implemented method for performing uncertainty-guided use of external sources of knowledge for response generation. The method includes obtaining, by a computing system comprising one or more computing devices, an input query. The method includes processing, by the computing system, the input query with a sequence processing model to generate a plurality of different responses to the input query as respective outputs of the sequence processingmodel. The method includes determining, by the computing system, whether the plurality of different responses satisfy a consensus metric. The method includes, in response to determining that the plurality of different responses do not satisfy the consensus metric: retrieving, by the computing system, one or more additional items of content from one or more external sources of knowledge; and processing, by the computing system, the input query and the one or more additional items of content with the sequence processing model to generate at least one additional response to the input query as an additional output of the sequence processing model. Other example aspects are directed to computer systems and non-transitory computer-readable media that are configured to perform or enable performance of the method.

[0009] Another example aspect of the present disclosure is directed to a computer- implemented method to facilitate processing of large medical context inputs with improved accuracy. The method includes obtaining, by a computing system comprising one or more computing devices, an input medical query and a set of medical context data. The method includes performing, by the computing system, a multi-stage chain-of-prompting approach to process the input medical query and the set of medical context data with a sequence processing model to generate an output response as an output of the sequence processing model. The method includes providing, by the computing system, the output response in response to the input medical query. Performing, by the computing system, the multi-stage chain-of-prompting approach comprises iteratively prompting, by the computing system, the sequence processing model with a series of prompts that are at least in part pre-defined. The series of prompts elicits, over the series of prompts, the sequence processing model to perform a chain of analysis on the input query and the set of medical context data to generate the output response. Other example aspects are directed to computer systems and non- transitory computer-readable media that are configured to perform or enable performance of the method.

[0010] Another example aspect of the present disclosure is directed to a computer- implemented method to facilitate processing of large medical context inputs with improved accuracy. The method includes obtaining, by a computing system comprising one or more computing devices, an input medical query and a set of multi-modal medical context data, wherein the set of multi-modal medical context data comprises data associated with at least two different data modalities. The method includes processing, by the computing system, the input medical query the set of multi-modal medical context data with a sequence processing model to generate one or more responses to the medical query as an output of the sequenceprocessing model. The sequence processing model comprises a decoder transformer model comprising a plurality of transformer blocks, wherein at least one of the plurality of transformer blocks comprises a sparse feed-forward-network layer comprises a mixture of experts networks. Other example aspects are directed to computer systems and non-transitory computer-readable media that are configured to perform or enable performance of the method.

[0011] Other aspects of the present disclosure are directed to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0012] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the related principles.

[0013] The attached Appendix, which is fully incorporated into and forms a portion of this disclosure, describes example implementations of the systems and methods described herein. The present disclosure is not limited to the example implementations described in the attached Appendix.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Detailed discussion of example embodiments directed to one of ordinary skill in the art is set forth in the specification, which makes reference to the appended figures, in which:

[0015] Figure 1 is a block diagram of an example self-training approach according to example implementations of aspects of the present disclosure;

[0016] Figure 2 is a block diagram of an example self-training approach according to example implementations of aspects of the present disclosure;

[0017] Figure 3 is flow chart diagram of an method to perform self-training according to example implementations of aspects of the present disclosure;

[0018] Figure 4 is a block diagram of an example uncertainty-guided search approach according to example implementations of aspects of the present disclosure;

[0019] Figure 5 is a flow chart diagram of an example method to perform uncertainty- guided search according to example implementations of aspects of the present disclosure;

[0020] Figure 6 is a flow chart diagram of an example method to perform chain-of- prompting for long-context medical inputs according to example implementations of aspects of the present disclosure;

[0021] Figure 7 is a block diagram of an example application of a decoder transformer with mixture of experts to perform processing of multi-modal medical data according to example implementations of the present disclosure;

[0022] Figure 8 is a flow chart diagram illustrating an example method for training a machine-learned model according to example implementations of aspects of the present disclosure;

[0023] Figure 9 is a block diagram of an example processing flow for using machine- learned model(s) to process input(s) to generate output(s) according to example implementations of aspects of the present disclosure;

[0024] Figure 10 is a block diagram of an example sequence processing model according to example implementations of aspects of the present disclosure;

[0025] Figure 11 is a block diagram of an example technique for populating an example input sequence for processing by a sequence processing model according to example implementations of aspects of the present disclosure;

[0026] Figure 12 is a block diagram of an example model development platform according to example implementations of aspects of the present disclosure;

[0027] Figure 13 is a block diagram of an example training workflow for training a machine-learned model according to example implementations of aspects of the present disclosure;

[0028] Figure 14 is a block diagram of an inference system for operating one or more machine-learned model(s) to perform inference according to example implementations of aspects of the present disclosure;

[0029] Figure 15 is a block diagram of an example networked computing system according to example implementations of aspects of the present disclosure;

[0030] Figure 16 is a block diagram of an example computing device according to example implementations of aspects of the present disclosure; and

[0031] Figure 17 is a block diagram of an example computing device according to example implementations of aspects of the present disclosure.DETAILED DESCRIPTION

[0032] The present disclosure provides a number of different systems and methods that enhance the performance and reliability of sequence processing models, particularly when applied to the processing of medical data and queries. The proposed techniques collectively address the challenges of integrating vast, evolving external knowledge sources, refining responses in the face of uncertainty, and efficiently managing complex, multi-modal datasets such as are commonly found in the medical field. Specifically, by leveraging iterative self-training, uncertainty-guided retrieval, multi-stage prompting, and / or advanced model architectures, the disclosed technology improves how machine learning models can process, analyze, and / or utilize complex information, resulting in improved computer systems applicable to a number of different applications or use cases, including medical diagnostics and / or other medical applications.

[0033] More particularly, one example aspect of the present disclosure is directed to an iterative self-training framework for teaching sequence processing models to leverage external sources of knowledge. This framework can use a combination of self-play and simulated data to teach the model how to effectively formulate information retrieval queries and integrate the retrieved information into its reasoning process. For instance, the model can learn to distinguish between different types of medical terminology and understand when it needs to search for definitions, symptoms, treatments, or research studies to answer a question accurately.

[0034] The proposed self-training framework can begin with the generation of a number of exemplar tuples which can be used as seeds for the process. In particular, each exemplar tuple can include an exemplar query and a corresponding exemplar response that relies upon and references one or more items of content from external sources. Specifically, the exemplar response can include a chain of analysis or reasoning that leads from the query to an exemplar answer to the query, where the chain of analysis relies upon or makes reference to the item(s) of content. In practice, this can include a human expert manually creating or otherwise curating an exemplar tuple. As one example, the query could ask about the side effects of a new medication, and the response could include information from the latest articles found on medical databases or trusted health websites. Thus, each exemplar tuple can represent a useful demonstration of how to create and follow a chain of analysis from a query to an answer while leveraging items of content from external sources of knowledge.

[0035] Next, a sequence processing model can be prompted to attempt to create additional synthetic data examples and / or demonstrate its ability to perform a chain of analysis. In particular, one or more exemplar tuples can be included as few-shot examples within a training prompt that is used to elicit the generation of a candidate synthetic response from the sequence processing model. In particular, the training prompt can include one or more of the exemplar tuple(s), a training query, and one or more additional training items of content obtained from external knowledge sources. The training query can be a different query than the exemplar query included in the exemplar tuple. Thus, a training prompt can be built that includes exemplar(s) that each demonstrate a chain of analysis leveraging external knowledge. The training prompt can also prompt the sequence processing model to create a new chain of analysis using new external knowledge.

[0036] In some implementations, the step of building the training prompt can include retrieving the training item(s) of content for inclusion in the training prompt. As one example, the training items of content included in the training prompt can be retrieved by processing the training query with the same sequence processing model or a different model to generate an information retrieval query (e.g., web search query, database query, etc.). For example, the model can be given the training query and can be prompted to generate an information retrieval query that will return relevant items of content. The generated information retrieval query can then be executed to retrieve the training items of content. As one example, this step can include using natural language processing techniques to transform a complex medical question into a simplified search query that can be used to fetch relevant articles or data from a web search engine or a specialized medical database.

[0037] The sources of external knowledge utilized in the present disclosure can vary, including search engines, the Internet, knowledge graphs, databases of domain-specific knowledge or information (e.g., medical knowledge or electronic medical or health records), application programming interface calls to knowledge sources, and / or various other sources of knowledge, data, or information which may, in some instances, be referred to as “tools”. For example, the model could use a general search engine to find information on common medical conditions or a specialized medical database to access peer-reviewed articles and clinical guidelines for more complex or rare diseases.

[0038] The sequence processing model can then process the full training prompt to generate a candidate response. Specifically, the candidate response can include a modelgenerated chain of analysis that relies upon or makes reference to the training items of content in arriving at a candidate answer to the training query. Thus, by processing thetraining prompt, the sequence processing model creates a synthetic candidate response that may include an additional demonstration of the use of external knowledge within a chain of analysis. The model-generated chain of analysis can demonstrate the model’s (current) ability to reason and arrive at a candidate answer to the training query by making use of the training items of content. For example, the model-generated chain of analysis could include a logical sequence of steps to diagnose a condition based on symptoms described in the training query and content sourced from medical journals.

[0039] Next, the candidate response generated by the sequence processing model in response to the training query can be evaluated against a ground truth answer to the training query. As one example, this evaluation can be conducted by comparing the candidate answer in the model-generated response to a verified answer to the training query. Thus, the ground truth answer can serve as a benchmark for correctness or accuracy. For example, the ground truth answer can be derived from reliable sources or expert knowledge within the relevant field. By determining the correspondence between the candidate response and the ground truth answer, the computing system can assess the model’s current performance level.

[0040] Further, this comparison can also be used to decide whether the candidate response should be used for further fine-tuning of the model or discarded to prevent reinforcement of incorrect reasoning. For example, if the candidate response correctly corresponds to the ground truth answer, then the model can be fine-tuned on the candidate response. However, if the candidate response does not correspond to the ground truth answer, then the candidate response can be discarded (e.g., deleted from memory) and the model is not further fine-tuned or trained on the candidate response.

[0041] Specifically, upon confirmation that a candidate response generated by the sequence processing model aligns with the ground truth answer, a subsequent step in the training process can include fine-tuning of the sequence processing model on this verified candidate response. Because the model is trained on its own generated responses, this process can be referred to as self-play or self-training. The fine-tuning process can include adjusting the model’s parameters to reinforce the successful reasoning paths and analytical strategies that led to the correct answer. The fine-tuning not only enhances the model’s ability to replicate expert-level reasoning for similar future queries but also contributes to the overall robustness of the model by expanding its knowledge base across newly-seen training prompts.

[0042] In some implementations, the process described above can be performed iteratively. By iteratively updating the model with these validated responses, the technologyensures continuous learning and improvement of the model’s accuracy and reliability. That is, when the model is fine-tuned on a validated response, the model’s ability to perform chain of analysis on external data sources can be improved. Therefore, by iteratively performing candidate response generation, validation, and fine-tuning, the model’s performance can be improved over time.

[0043] Furthermore, in some implementations, when a candidate response is validated (e.g., confirmed to correspond to or include the ground truth answer) the candidate response can optionally be added to the corpus of exemplars. For example, the candidate response may thereafter serve as an exemplar within a training prompt in a subsequent round of selftraining. For instance, successful responses to queries about emergency medical procedures can be added to the exemplar set to improve the model’s future performance on similar topics.

[0044] Conversely, the present disclosure includes a mechanism for discarding candidate responses that do not correspond to the ground truth answer. This helps reduce a likelihood that the model is reinforced by incorrect information. For example, if the model incorrectly identifies a symptom as indicative of a particular disease, that response is discarded to help prevent future misdiagnoses.

[0045] The proposed self-training framework provides a number of technical effects and benefits. As one example, the disclosed method for self-training sequence processing models to utilize external sources of medical knowledge provides a technical solution to a concrete technical problem within the medical field. This method actively engages a specific technical system — the sequence processing model — to interpret complex medical data, yielding accurate and reliable medical diagnoses.

[0046] As another example, the systems and methods of the present disclosure are specifically adapted to technical implementations. In particular, the disclosed iterative selftraining framework, which instructs sequence processing models to incorporate web-search- derived information into their analytical processes, represents a specific technical framework to integrate a machine learning model with external sources of knowledge. This framework is tailored to the internal operations of the computational system, for example encompassing both the sequence processing model and external tool interfaces (e.g., a web search API). The integration of the sequence processing model with external sources is driven by technical considerations, such as the efficient assimilation and processing of extensive data from diverse external sources.

[0047] As another example technical effect, the training methodology set forth in this disclosure results in technical effects that improve the functioning of the sequence processing model and of the computing system executing the sequence processing model. In particular, the ability of a computer system to retrieve and analyze content from external sources of knowledge is improved.

[0048] Another example aspect of the present disclosure is directed to systems and methods to enhance the reliability of responses generated by a sequence processing model, particularly when dealing with complex queries such as medical questions. Example implementations of this approach can be referred to as uncertainty-guided search or information retrieval at inference. Thus, the proposed uncertainty-guided approach can be performed at inference time to assist in generating a response to an input query.

[0049] Specifically, some example implementations can include obtaining an input query and processing it to produce multiple different responses. For example, when a user inputs a medical question, the system may generate several diagnostic suggestions or treatment options as responses.

[0050] In some implementations, each of the responses generated by the system includes a chain of analysis that leads from the input query to the respective answer. This chain can be a logical reasoning path that explains how the system arrived at the answer. Providing a rationale can be helpful in fields where understanding the rationale behind a decision is important, such as in medicine or law.

[0051] The multiple different generated responses can be evaluated to determine whether they satisfy a consensus metric. For example, this can be done through various techniques such as comparing the similarity of responses or using statistical methods to measure the agreement among the responses. For example, in a medical context, if multiple responses suggest the same diagnosis, the system may determine that there is a consensus. As one example, determining whether there is a consensus among the different responses can include calculating the entropy (e.g., Shannon’s entropy) of the distribution of answers. This can be compared to a threshold value to assess uncertainty. For example, a high entropy value, indicating a wide spread of different responses, can indicate that there is a lack of consensus, which in turn can trigger the system to seek additional information to reduce uncertainty.

[0052] In particular, when the responses demonstrate consensus, then a consensus answer from the responses can be provided as an output. However, when the responses do not meet the consensus metric, indicating potential uncertainty or conflict in the information, thepresent disclosure can operate to access or otherwise obtain additional information from external sources of knowledge to further inform the response generation process. For example, this can include querying medical databases, scientific journals, or the wider web or Internet using search engines to find relevant information that can clarify the uncertainty.

[0053] Furthermore, in cases where the initial responses exhibit a conflict, the system can identify these conflicting responses and generate a targeted information retrieval query. This query can be designed to identify and return content that addresses the specific point of conflict. For example, if there is a disagreement about the side effects of a medication, the system can generate a query to find scientific studies that discuss those specific side effects.

[0054] In some implementations, the information retrieval query is crafted by prompting the sequence processing model, or another model, with the conflicting responses and instructions to generate the query. This ensures that the search is focused and relevant to the specific uncertainty at hand. In practice, this can include using natural language processing techniques to formulate questions or structured queries that are likely to retrieve the most informative results from a medical database.

[0055] The items of content retrieved from the external sources of knowledge can then added to or otherwise used to update the original input and the sequence processing model can again generate a plurality of additional responses. These additional responses can be informed by (e.g., rely upon and / or cite to) the newly-retrieved items of content. For example, the system may re-evaluate a medical question in light of recent clinical guidelines found in the additional content, or it may incorporate statistical data from recent studies to provide a more evidence-based answer.

[0056] Then, the system can again determine whether the additional responses, now informed by the external content, satisfy the consensus metric. Similarly to the previous iteration, if the additional responses demonstrate consensus, then a consensus answer from the additional responses can be provided as an output. For example, this could be implemented by presenting the most common response or a model -synthesized response that incorporates elements from each of the agreeing responses. For instance, if multiple sources suggest a particular medication for a symptom, the system can output that medication as the recommended treatment. However, if the additional responses do not meet the consensus metric, indicating remaining uncertainty or conflict in the information, the present disclosure can operate to again access or otherwise obtain yet additional information from the external sources of knowledge.

[0057] Thus, in some implementations, the uncertainty-guided process of retrieving additional items of content is iterative and can include several iterations. For instance, after the initial responses are generated, the system can access a medical database to find articles related to the symptoms described in the input query. The information from these articles is then used to refine the responses generated in a subsequent iteration. If consensus is still not reached, then yet further information can be retrieved and used to again seek to identify a consensus answer, and so on.

[0058] The external sources of knowledge utilized by the system can include a variety of platforms, such as search engines and the Internet, or more specialized databases. For medical queries, the system may access databases containing medical knowledge or records, retrieving articles or records that can aid in generating a more informed response.

[0059] The proposed uncertainty-guided framework provides a number of technical effects and benefits. As one example, the disclosed method can enhance the functionality and reliability of medical diagnostic systems by ensuring that the responses generated are factually accurate and dependable. In healthcare contexts, where precision is of high importance, the method’s ability to iteratively refine responses using external medical knowledge sources directly contributes to improving patient outcomes. By incorporating the latest medical research and data into the reasoning process, the method provides a practical solution to the technical problem of outdated or incomplete medical information, which can be beneficial in clinical decision-making scenarios.

[0060] As another example technical effect, the proposed method exhibits a specific technical implementation of iterative enhancement of query responses. Leveraging a sequence processing model, the method systematically incorporates external data sources into the reasoning process, thereby refining the accuracy of its outputs. By doing so, it addresses the technical challenge of integrating vast and dynamic medical knowledge bases with artificial intelligence systems.

[0061] The proposed uncertainty-guided framework also optimizes computational efficiency within question-answering systems. By employing an uncertainty-guided search strategy, the method selectively processes information until consensus is reached (but not beyond), thereby reducing unnecessary computational load and improving the speed and accuracy of responses. More particularly, the proposed technique specifically selects and processes content items with the objective of resolving specific points of conflict or uncertainty to achieve consensus. Traditional systems, which do not measure uncertainty, may indiscriminately include and process a large volume of content, much of which may beirrelevant or superfluous for resolving a particular query. This traditional approach not only expends excessive computational resources but also extends query processing latency, and thus also the time required to reach a conclusive decision. In contrast, the proposed method determines points of uncertainty and iteratively refines the search and processing of content, thereby minimizing the amount of context processed. By focusing only on pertinent information and discontinuing the process upon reaching consensus, the method substantially reduces computational waste. Specifically, by pinpointing and addressing specific conflicts in reasoning, the system is able to reach consensus more rapidly, further conserving computational efforts and enhancing the overall efficiency and speed of medical diagnostic systems.

[0062] Another aspect of the present disclosure is directed to systems and methods for processing extensive medical data using a multi-stage chain-of-prompting approach. As an example, a computing system can obtain an input medical query and a set of medical context data, and then employing the multi-stage chain-of-prompting approach with a sequence processing model to generate a response. For example, the computing system can handle queries related to various medical conditions and produce accurate outputs by analyzing large sets of EHRs, genomic sequences, and / or medical imagery.

[0063] In some implementations, the sequence processing model used in the present disclosure can be fine-tuned for medical applications, enhancing its ability to understand and process complex medical data. For instance, medically fine-tuned LLMs or LMMs can be leveraged. Example models include models that have been trained to process million(s) of tokens and are capable of in-context learning.

[0064] The multi-stage chain-of-prompting approach enables the processing of medical queries with high precision. This approach can include iteratively prompting the sequence processing model with a series of at least partially pre-defined prompts that guide the model through a chain of analysis. For example, the series of prompts can include a series of prompt templates which are iteratively completed (e.g., filled in) with prior context and / or response content output by the model in response to prior prompt(s) in the series.

[0065] As one example, a first prompt can direct the model to retrieve relevant information from a vast set of medical data, while subsequent prompts could instruct the model to analyze the retrieved information for decision-making. In particular, in some implementations, a first prompt in the series of prompts can instruct the sequence processing model to perform information retrieval from a massive set of medical context data. This step helps to narrow down the vast amount of data to a smaller, more manageable set of retrieveddata, which is more relevant to the medical query at hand. For example, the model can be prompted to extract all mentions of a specific medical condition from a patient’s EHR. Following the initial retrieval, the series of prompts can include a second prompt that instructs the sequence processing model to perform a medical analysis on the smaller set of retrieved data. This step can include evaluating the relevance and implications of the retrieved mentions, which may include determining the presence of a disease, condition, or procedure. For instance, the model can first retrieve and then second analyze mentions of “hypothermia” across EHR notes to conclude whether a patient has a history of the condition.

[0066] In some implementations, the proposed multi-stage chain-of-prompting approach can handle medical queries related to specific diseases, conditions, or procedures. The queries can be diagnostic in nature, and the output response generated by the model can be an output diagnosis. For example, a clinician could input a query about a rare genetic disorder, and the model would process relevant EHRs and genomic data to provide a diagnosis.

[0067] In some implementations, the set of medical context data in the present disclosure can include data from various modalities, such as text data from EHRs, genomic sequences, and image data from medical imaging modalities. This multimodal approach allows for a comprehensive analysis of a patient’s medical history and current condition. For instance, the model could analyze both the textual description of a tumor from an EHR and the corresponding radiological images. As another example, multimodal processing capabilities are particularly useful for tasks such as generating radiology reports or annotating medical images, where the model can compare and integrate findings from textual and visual data.

[0068] In some implementations, the set of context data can include EHRs. The ability to analyze EHRs is helpful for retrieving information from a patient’s medical history, which can be extensive and complex. For example, the model can process hundreds of EHR notes to identify and confirm the presence of a particular medical condition.

[0069] The long-context processing capabilities enabled by the chain-of-prompting approach also enable the proposed systems to be applied to a number of different tasks. In particular, the proposed techniques enable the efficient processing of large amounts of data, thus reducing the risk of information overload for clinicians. The technology can assist in various clinical workflows, such as identifying rare conditions in lengthy EHRs or annotating complex medical videos, thereby enhancing the efficiency and effectiveness of patient care.Thus, the system can be applied to “needle-in-a-haystack” tasks, where the model must identify rare mentions of medical conditions within extensive EHRs or videos.

[0070] The long-context processing capabilities enabled by the chain-of-prompting approach also allow the proposed systems to be applied to a number of different tasks beyond the medical field. For example, in the context of web search, the technology can assist users in finding specific information within vast collections of web pages, effectively acting as a "needle-in-a-haystack" tool to extract precise data from the clutter of the internet. In library science, the system can be employed to sift through extensive digital archives to locate rare documents or references that are critical for academic research. For long-form video or audio content, such as films or podcasts, the technology can rapidly pinpoint moments or segments that contain particular topics or dialogue, thereby saving substantial time in content analysis. Similarly, within an email inbox that contains thousands of messages, the system can quickly surface emails pertaining to a specific subject or sender, streamlining the task of managing digital communications. These applications demonstrate the versatility and adaptability of the chain-of-prompting approach in handling large amounts of data across diverse domains, providing users with powerful tools for information retrieval and content management.

[0071] The proposed multi-stage chain-of-prompting approach provides a number of technical effects and benefits. As one example technical effect and benefit, the technology facilitates the sophisticated control and enhancement of complex technical systems, particularly for medical diagnostics. By leveraging advanced machine learning methods, the technology can process vast amounts of medical data with increased precision, leading to the more accurate identification of conditions with less time and repetition, thus contributing to the efficiency and effectiveness of medical diagnostic systems. This practical application of machine learning methods to real-world technical operations not only improves the functionality of such systems but also offers tangible improvements in patient outcomes and healthcare delivery.

[0072] As another example technical effect and benefit, the multi-stage chain-of- prompting approach disclosed herein significantly enhances the computational efficiency of the system by enabling focused processing over a carefully selected subset of medical context data. This targeted approach is achieved through the application of later prompts in the series, which guide the sequence processing model to refine the scope of its analysis after an initial broad retrieval. As a result, the system is not burdened with performing long-form text generation over an input that includes extraneous data, leading to a reduction in the computational resources required and an increase in processing speed. For example,generating responses based on large amounts of context can include attending over the context for each generation cycle (e.g., token prediction). By using a preliminary query to reduce the amount of context, one or multiple subsequent queries can be processed with smaller amounts of context, thereby decreasing the cost of computing responses to those one or multiple subsequent queries. This efficient data handling is particularly advantageous in medical applications where rapid and accurate analysis of large datasets is helpful for effective patient care and diagnostics.

[0073] Another example aspect of the present disclosure is directed to the use of long- context multi-modal sequence processing models to process multi-modal medical data and queries with improved accuracy. This technology can be implemented in various healthcare settings, such as hospitals, research institutions, and telemedicine platforms. By obtaining an input medical query and a set of multi-modal medical context data, the computing system can process this information using a multi-modal sequence processing model. For instance, a clinician can input a query regarding a patient’s symptoms, and the system can analyze related radiographs, lab results, and electronic health records to generate a response.

[0074] In some implementations, example sequence processing models within the present disclosure can include a decoder transformer model, which can include multiple transformer blocks. Some or all of these transformer blocks can include a sparse feed- forward-network layer with a mixture of experts networks, enabling the model to handle processing of large amounts of complex context data.

[0075] In particular, the decoder transformer model described herein can be designed to enhance the processing of complex context data by incorporating a novel architecture that leverages the power of sparsity and expert knowledge. The model can include multiple transformer blocks, each potentially equipped with a sparse feed-forward-network layer that integrates a mixture of experts networks. This configuration allows the model to selectively activate different subsets of parameters, or “experts,” depending on the specific input example being processed. By doing so, the model can scale its parameter space (e.g., enabling greater world knowledge and understanding capability) while only using a subset of its parameters at inference time, thereby at least partially decoupling overall capability from inference cost. The sparse nature of the feed-forward-network layers can engage only the most relevant experts for any given input, thereby optimizing efficiency and performance.

[0076] The decoder transformer model can be applied to applications including extremely long-context medical data, which is often characterized by complex patterns and relationships spanning extensive sequences of clinical notes, patient histories, and medicalliterature. The integration of sparse feed-forward-network layers with a mixture of experts networks within the model’s architecture allows for the dynamic allocation of computational resources to the most salient features within such lengthy and complex data. In the medical context, different experts can be specialized to recognize and process distinct types of medical information, such as symptoms, diagnoses, test results (in various modalities), treatment responses, temporal patient progression, etc.

[0077] For example, one expert within the mixture can become adept at interpreting radiology reports, while another can focus on longitudinal patient data, and yet another could specialize in pharmacological interactions. As the model encounters various segments of the long-context medical data, the sparse activation mechanism can engage only the most relevant expert(s), thus enabling the model to maintain high performance across diverse datatypes in a single, voluminous input. This results in a highly efficient and effective analysis of medical data, where the model can provide insights into patient outcomes, suggest personalized treatment plans, and even aid in the discovery of complex medical correlations that may not be immediately apparent to human practitioners.

[0078] In some implementations, the input medical query can relate to a specific disease, condition, or procedure. This allows for targeted analysis and response generation by the computing system. For instance, a query could focus on identifying the presence of a particular cancer type, understanding the progression of a chronic disease, or determining the most appropriate surgical procedure for a patient’s condition.

[0079] In some implementations, the set of multi-modal medical context data can include data associated with at least two different data modalities. This can include a combination of image data, such as ultrasound or mammography images, and text data, including patient history and symptoms documentation. By integrating these different data types, the system can provide a more holistic view of the patient’s condition.

[0080] For example, in some implementations, the multi-modal medical context data can include radiograph data. This can be useful in fields such as radiology and orthopedics, where imaging plays an important role in diagnosis and treatment planning. For example, the technology can be used to analyze a series of X-rays to monitor the healing progress of a bone fracture.

[0081] As another example, in some implementations, the multi-modal medical context data can include genomic data, which can be beneficial for personalized medicine and genetic research. For instance, the system can process a genomic sequence data to assess the risk of hereditary diseases or to tailor a treatment plan based on genetic markers.

[0082] As another example, in some implementations, the multi-modal medical context data can include chemical or molecular data. This capability can be advantageous in pharmacology and toxicology to help understand the interactions between various compounds and biological systems. An example application could be predicting the efficacy of a new drug compound based on its molecular structure and patient-specific factors.

[0083] As another example, in some implementations, the multi-modal medical context data can include a plurality of electronic health records. An example implementation can assist with long-term patient care and population health management. For example, the system can analyze a patient’s entire medical history, including previous diagnoses, treatments, and outcomes, to inform current clinical decisions.

[0084] In some implementations, the input medical query includes a diagnostic query, and the output response includes an output diagnosis. This feature can be particularly helpful for emergency medicine or remote diagnostics. For instance, the system can receive a query about a set of symptoms indicative of a heart attack and rapidly analyze relevant data to confirm the diagnosis and suggest immediate interventions.

[0085] The proposed system and capabilities can be integrated into existing medical information systems, enhancing their functionality. For instance, a hospital’s electronic medical record system can be updated with this technology to improve the accuracy and comprehensiveness of the medical data analysis it provides to healthcare providers.

[0086] The proposed techniques represent a tangible improvement in the field of medical data processing through the use of long- context multi-modal sequence processing models. These models, featuring advanced decoder transformer architectures with a mixture of experts networks, enable the efficient and accurate analysis of multi-modal medical data, including text and images. This technical advancement directly addresses the challenge of processing complex and voluminous medical datasets by offering a method that significantly enhances the computational efficiency and accuracy of medical diagnostics. By facilitating the integration of diverse data types and providing comprehensive responses to medical queries, the disclosed technology not only improves the functionality of medical Al systems but also contributes to the technical progress in the healthcare industry.

[0087] The technology disclosed herein can be applied to a variety of use cases within the medical field, where precision and up-to-date information are important. For example, in a clinical decision support system, a physician could input a complex patient query regarding a potential drug interaction between two specific medications. The sequence processing model, utilizing its training, could perform a web search to retrieve the most current researchon the medications in question. The output from the model would be a detailed explanation of the interaction for consideration by the physician, including any known side effects or contraindications, which is informed by the latest medical literature. In another example, the model could be used in telemedicine platforms, where a patient’s symptoms are inputted, and the model searches for similar case studies or clinical guidelines, outputting a probable diagnosis and recommended next steps for treatment for consideration by a medical professional. As yet another example, the model could be used in medical education, where a student inputs a hypothetical patient scenario, and the model outputs a step-by-step clinical reasoning path, including references to relevant medical texts and research articles, to aid in the learning process.

[0088] As further examples, the disclosed systems and methods present a framework that can be applied to control and enhance various technical systems or processes within the medical field, such as diagnostic imaging equipment, robotic surgery apparatuses, and patient monitoring devices. By incorporating the self-training, uncertainty-guided retrieval, multistage prompting, and / or multi-modal processing capabilities of advanced sequence processing models, these systems can autonomously adjust operational parameters, refine diagnostic outputs, and optimize patient care protocols. For example, an imaging device could utilize these methods to tailor its scanning technique based on a patient’s unique medical history processed by the model, while a robotic surgery system can adjust its operative plan by analyzing real-time surgical data against a vast database of surgical outcomes. This knowledge-driven approach ensures that medical devices operate with enhanced precision, efficiency, and alignment with the latest medical standards, ultimately contributing to better patient outcomes and more streamlined healthcare workflows.

[0089] As yet further examples, the disclosed technology can enhance the capability of healthcare systems in generating radiology reports, summarizing medical texts, composing referral letters, and conducting multimodal diagnostic dialogues. In radiology report generation, the system can process input modalities such as radiographic images and accompanying clinical notes to output structured reports that detail findings, comparisons, and recommendations in a standardized format. For medical text summarization, the system can receive extensive medical literature or patient records as inputs and generate concise summaries that capture the semantic meaning of the content, aiding in quick information assimilation. The creation of referral letters can be streamlined by inputting a patient's medical history and current condition to produce tailored referral communications that efficiently convey relevant information to specialists. Furthermore, the system can facilitatemultimodal diagnostic dialogues by integrating inputs from both textual patient interactions and associated medical imagery or test results, with outputs comprising comprehensive dialogue exchanges that support diagnosis and treatment planning. Each of these applications leverages the system's ability to synthesize and contextualize large volumes of complex data.

[0090] Outside the medical sphere, the disclosed technology can be applied to various non-medical use cases where access to and integration of external information are beneficial. For example, in the legal industry, a legal professional could input a query regarding a specific point of case law; the model would then search legal databases to provide the most recent and relevant legal precedents and statutes, outputting a synthesized legal argument or memorandum. As another example, the technology can be utilized in customer service agents, where a customer’s inquiry about a product feature or issue is inputted, and the model outputs a tailored solution or guidance by searching through product manuals, FAQs, and customer forums for similar queries and resolutions. As another example, the disclosed technology could be used as a search engine assistant or aggregator. For example, a user could input a search query. The system could retrieve a number of web results and then provide the user with a chain of analysis that provides an answer to the query while relying upon and making reference to the web results.

[0091] Various example implementations are described herein with respect to the accompanying Figures.Example Self-Training

[0092] Referring now to FIG. 1, a schematic block diagram is depicted which illustrates an embodiment of the iterative self-training process for a sequence processing model 122 in accordance with the disclosed technology. The process begins with the creation of an exemplar tuple 102, which includes an exemplar query 104 and a corresponding exemplar response 106. The exemplar response 106 comprises an exemplar chain of analysis 108 that culminates in an exemplar answer 110. This exemplar chain of analysis 108 relies upon or references one or more exemplar items of content obtained from external sources of knowledge 112.

[0093] The computing system then generates a training prompt 114, which includes the exemplar tuple 102, a training query 116, and one or more training items of content 118. These training items of content 118 are also obtained from the external sources of knowledge 112 and are accompanied by instructions 120 for the sequence processing model 122 to generate a candidate response 124. Upon processing the training prompt 114, the sequenceprocessing model 122 outputs the candidate response 124, which includes a model -generated chain of analysis 126 leading to a candidate answer 128.

[0094] A correctness evaluator 130 then assesses whether the candidate answer 128 corresponds to a ground truth answer 131 to the training query 116. For example, the correctness evaluator can itself be a machine learning model or may some other heuristic embodied in computer-executable instructions stored on a computer-readable medium.

[0095] If the candidate response 124 is determined to be correct, the model trainer 132 fine-tunes the sequence processing model 122 on the candidate response 124.Conversely, if the candidate response 124 is incorrect, a discard operator 134 removes the candidate response 124 from consideration. In such fashion, incorrect reasoning patterns are not reinforced. The model trainer 132 and discard operator 134 can include computerexecutable instructions stored on a computer-readable medium.

[0096] For example, the exemplar query 104 could be “What are the potential side effects of medication X?” and the exemplar response 106 can detail side effects based on the latest research from medical databases, which are part of the external sources of knowledge 112. The training query 116 can then pose a different question related to medication X, prompting the sequence processing model 122 to use the provided data and its own reasoning to generate a new, accurate response informed by the most recent and relevant medical information.

[0097] In some implementations, the process can be iterative, with the sequence processing model 122 being continuously updated based on validated responses. This iterative process ensures the model’s ongoing improvement and adaptation to new data and queries.

[0098] Referring now to FIG. 2, a schematic representation illustrates the fine-tuning loop of a sequence processing model 200 in accordance with the disclosed technology. The sequence processing model 200 is depicted as receiving two types of inputs: a training prompt without search results 202 and a training prompt with search results 204. These inputs are indicative of the model’s capability to process and analyze queries with varying levels of available information, thereby simulating real-world scenarios where certain data may or may not be readily accessible.

[0099] The sequence processing model 200 processes these inputs to generate respective candidate responses, which are then subjected to a scoring and filtering process 206. This process evaluates the generated responses against predefined criteria to ensure their relevance and accuracy. Candidate responses that meet the criteria are funneled into the fine-tuning loop 208, where the sequence processing model 200 is iteratively updated to improve its performance.

[0100] Within the fine-tuning loop 208, candidate responses can be represented as chains of analysis (also referred to as “chains of thought” (CoT)) with corresponding answers (Answer). These responses are either based solely on the content of the training prompt (e.g., CoT) or enriched with information retrieved from search results (e.g., CoT with search). This differentiation highlights the model’s ability to adapt its analysis based on the availability of external information, which is important for maintaining the robustness and flexibility of the system.

[0101] For example, the training prompt without search results 202 can contain a medical query that the sequence processing model 200 attempts to answer based on its preexisting knowledge and training. In contrast, the training prompt with search results 204 could include additional data retrieved from medical databases or journals, which the model integrates into its reasoning process to formulate a more informed response.

[0102] The fine-tuning loop 208 represents an iterative learning process where the sequence processing model 200 is continuously refined through exposure to a diverse range of queries and information sources. This iterative process can include the incorporation of validated responses into the model’s training set, allowing it to learn from its successes and further enhance its analytical capabilities over time.

[0103] In some implementations, the scoring and filtering process 206 can employ machine learning algorithms to assess the quality of the candidate responses. In such fashion, only the most accurate and relevant responses contribute to the fine-tuning of the sequence processing model 200. Additionally, alternative embodiments can include human experts in the scoring process to provide a layer of domain-specific validation, particularly for complex medical queries where nuanced understanding is important.

[0104] Referring now to FIG. 3, a flowchart 300 illustrates an embodiment of the iterative self-training process for a sequence processing model in accordance with the disclosed technology. The process initiates with the step 302 of obtaining an exemplar tuple, which serves as an exemplary learning instance for the model. This tuple includes an exemplar query and an associated exemplar response that demonstrates how to utilize external knowledge sources in formulating an answer.

[0105] Subsequently, at 304, the computing system generates a training prompt, which encompasses the exemplar tuple, a novel training query, and additional training items of content that are sourced from external knowledge repositories. These componentscollectively form a comprehensive context for the sequence processing model to interpret and learn from.

[0106] The sequence processing model is then tasked with processing at 306 the training prompt to produce a candidate response. This response can include a modelgenerated output that includes a detailed chain of analysis leading to a potential answer to the training query, illustrating the model’s current reasoning capabilities.

[0107] An evaluation step follows at 308, where the computing system determines whether the candidate response corresponds to a ground truth answer to the training query. This determination assesses the accuracy and reliability of the model’s output.

[0108] If the candidate response is deemed to align with the ground truth, then at 310 the model undergoes a fine-tuning process on this validated response, which serves to reinforce correct analytical patterns and enhance the model’s performance.

[0109] Optionally, at 312, the validated candidate response can be added to a corpus of exemplars. This addition serves to enrich the training dataset and provide future prompts with a more diverse range of examples to learn from.

[0110] Alternatively, if the candidate response does not match the ground truth, it is discarded 314 to prevent the reinforcement of erroneous reasoning within the model.Example Uncertainty-Guided Information Retrieval

[0111] Referring now to FIG. 4, a schematic diagram illustrates an embodiment of the uncertainty -guided search loop utilized by a sequence processing model 400 in accordance with the disclosed technology. The sequence processing model 400 receives an input prompt that includes an instruction, a query, and search results. Upon processing this input, the model generates multiple chains of thought (CoT), each leading to a potential answer to the query. These CoTs, represented as CoT #2 402 leading to Answer A and CoT #n 406 leading to Answer B, illustrate the model’s ability to analyze the input from various perspectives and propose different solutions or answers.

[0112] The diagram further depicts a decision point, querying whether a consensus has been reached among the generated CoTs. If the answer is affirmative, the model outputs a consensus CoT, providing a unified answer, e.g., Answer A, which is considered the most reliable response based on the agreement of multiple CoTs. However, if no consensus is reached, the model engages in an uncertainty-guided search loop.

[0113] This loop includes the sequence processing model 400 generating targeted search queries to resolve conflicts identified among the CoTs, such as the conflicting CoT(Answer A) 402 and CoT (Answer B) 406. The model then conducts a web search to retrieve additional context, which can either support Answer A or refute Answer B, or vice versa. The newly obtained context is then fed back into the sequence processing model 400, which processes the enriched input to generate additional CoTs and, potentially, reach a consensus.

[0114] For example, if the initial CoTs provide conflicting information about the side effects of a medication, the sequence processing model 400 generates search queries specifically designed to find authoritative medical literature that clarifies the discrepancy. The additional context obtained from the web search may include recent clinical trial results or expert medical opinions that support or challenge the initial answers provided by the model.

[0115] The uncertainty-guided search loop exemplified in FIG. 4 can be an iterative process that refines the model’s outputs by incorporating external knowledge, thus improving the reliability and accuracy of the responses. This approach ensures that the sequence processing model 400 remains dynamic and adaptable, capable of resolving uncertainties through targeted information retrieval and analysis.

[0116] Referring now to FIG. 5, a flowchart 500 is provided illustrating an embodiment of the uncertainty-guided search process utilized by a sequence processing model in accordance with the disclosed technology. The process commences with an operation 502 where an input query is obtained by a computing system. This input query can be of a medical nature, such as a complex patient symptom inquiry or a request for information on drug interactions.

[0117] Following the receipt of the input query, the sequence processing model proceeds to operation 504, where it processes the input query alongside available items of content to generate a plurality of different responses. For example, the model may produce varied diagnostic suggestions or treatment plans based on the input medical query. Each of these responses can represent a unique chain of analysis derived from the input query and the content with which the model is provided.

[0118] At operation 506, the computing system determines whether the generated responses satisfy a consensus metric. This metric assessment can include comparing the similarity of responses or utilizing statistical methods to evaluate the agreement among the responses. For instance, if the responses consistently suggest a particular diagnosis, the system may deduce that a consensus has been achieved.

[0119] If the consensus metric is satisfied, as indicated by the affirmative path leading to operation 508, the computing system provides at least a portion of at least one of theplurality of responses as an output. This output can be the most common response among the plurality or a synthesized response that incorporates elements from the agreeing responses.

[0120] Conversely, if the metric is not satisfied, indicating potential uncertainty or conflicting information within the responses, as indicated by the negative path leading to operation 510, the system identifies two or more responses that exhibit a conflict.Subsequently, at operation 512, an information retrieval query is generated that is specifically designed to return items of content related to the identified conflict. For instance, if there is a discrepancy in the suggested side effects of a medication among the responses, the query generated can be aimed at retrieving scientific studies discussing those specific side effects.

[0121] Finally, at operation 514, the system queries one or more external sources of knowledge with the information retrieval query to obtain additional items of content. These external sources can include medical databases, scientific journals, or the wider web. The newly retrieved items of content are then used to update the original input, and the sequence processing model generates additional responses that are informed by this new content, potentially leading to a consensus.

[0122] In some implementations, the operations depicted in FIG. 5 can be performed iteratively, with the sequence processing model refining its responses through successive rounds of uncertainty-guided search and analysis. This iterative process can include several layers of conflict identification, targeted information retrieval, and response generation until a consensus is reached or until the system determines that further searching would not result in a consensus.Example Chain-of-Prompting for Long-Context Inputs

[0123] Referring now to FIG. 6, a flowchart 600 is provided to illustrate an embodiment of a multi-stage chain-of-prompting process for a sequence processing model in accordance with the disclosed technology. The process begins at operation 602, where an input medical query and a set of medical context data are obtained by a computing system.This set of medical context data can include, for example, electronic health records, genomic data, or radiological images, providing a comprehensive dataset for analysis.

[0124] At operation 604, the sequence processing model is prompted with a current prompt from a series of at least partially pre-defined prompts. These prompts guide the sequence processing model through a structured chain of analysis which may, in some implementations, be tailored to the medical query and the medical context data. For instance,the prompts may direct the model to focus on specific aspects of the data, such as identifying symptoms, reviewing patient history, or analyzing test results.

[0125] Upon receiving the current prompt, the sequence processing model generates a response at operation 606. This response is a product of the model’s processing capabilities, which can include advanced natural language understanding and multi-modal data integration, to provide a reasoned and contextually relevant output.

[0126] The process then proceeds to decision operation 608, where it is determined whether additional prompts remain in the series of prompts. If additional prompts are present, indicating that the chain of analysis-based prompting is not yet complete, the process moves to operation 612. Here, a new prompt is generated based on the prior response(s) and the template prompt contained in the series of prompts. This iterative prompting continues, with each new prompt building upon the information and analysis provided in the previous responses, thereby refining the output response progressively.

[0127] If no additional prompts remain, as determined at operation 608, the process concludes with operation 610, where the output response is provided. This output response represents the culmination of the multi-stage chain-of-prompting process, offering a comprehensive and precise answer to the input medical query based on the extensive analysis of the medical context data.

[0128] The multi-stage chain-of-prompting process depicted in FIG. 6 exemplifies the iterative and dynamic capabilities of the disclosed technology in processing complex medical queries. By leveraging a series of structured prompts, the sequence processing model can systematically dissect and analyze large volumes of medical data, leading to accurate and actionable outputs that can significantly aid in medical diagnostics and decision-making.

[0129] In some implementations, the multi-stage chain-of-prompting process can be adapted to various medical applications, such as generating differential diagnoses, recommending treatment plans, or conducting medical research. The flexibility of the process allows for customization of the prompts to suit specific medical scenarios, thereby enhancing the utility and applicability of the sequence processing model in diverse healthcare settings.Example Long-Context Models for Multi-Modal Medical Processing

[0130] Referring now to FIG. 7, a schematic diagram is presented, illustrating an advanced architecture for a sequence processing model adapted for handling multi-modal medical context data. The depicted model is structured to enhance the processing of complex datasets. At the core of the system is a decoder transformer 706, which can include multipletransformer blocks, such as transformer block 708, transformer block 710, and transformer block 712. These blocks are designed to sequentially process an input medical query 702 and multi-modal medical context data 704, leading to the generation of a response 714.

[0131] Some or all of the transformer blocks, e.g., exemplified by transformer block 712, can include a sparse feed-forward-network layer that features a mixture of experts networks, as highlighted in the detailed view 716. This design enables selective activation of parameters or “experts” based on the input data. For instance, the detailed view 716 illustrates the possible routing of inputs, xl and x2, through different feed-forward-networks (FFN1, FFN2, FFN3, FFN4) with associated probabilities (p=0.65, p=0.8), which are governed by routers within the layer. This probabilistic approach to selecting experts allows the model to dynamically allocate computational resources to the most relevant expert network within the layer, optimizing both efficiency and performance.

[0132] In some implementations, the sequence processing model can be fine-tuned for specific medical applications, such as the analysis of radiograph data, genomic sequences, or chemical and molecular data. By leveraging the multi-modal capabilities of the model, healthcare professionals can obtain a more comprehensive understanding of a patient’s condition, leading to more accurate diagnoses and personalized treatment plans.Example Training Methods

[0133] Figure 8 depicts a flowchart of a method 800 for training one or more machine-learned models according to aspects of the present disclosure. For instance, an example machine-learned model can include a sequence processing model.

[0134] One or more portion(s) of example method 800 can be implemented by a computing system that includes one or more computing devices such as, for example, computing systems described with reference to the other figures. Each respective portion of example method 800 can be performed by any (or any combination) of one or more computing devices. Moreover, one or more portion(s) of example method 800 can be implemented on the hardware components of the device(s) described herein, for example, to train one or more systems or models. Figure 8 depicts elements performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, combined, or modified in various ways without deviating from the scope of the present disclosure. Figure 8 is described with reference to elements / terms described with respect to other systems and figures forexemplary illustrated purposes and is not meant to be limiting. One or more portions of example method 800 can be performed additionally, or alternatively, by other systems.

[0135] At 802, example method 800 can include obtaining a training instance. A set of training data can include a plurality of training instances divided between multiple datasets (e.g., a training dataset, a validation dataset, or testing dataset). A training instance can be labeled or unlabeled. Although referred to in example method 800 as a “training” instance, it is to be understood that runtime inferences can form training instances when a model is trained using an evaluation of the model’s performance on that runtime instance (e.g., online training / learning). Example data types for the training instance and various tasks associated therewith are described throughout the present disclosure.

[0136] At 804, example method 800 can include processing, using one or more machine-learned models, the training instance to generate an output. The output can be directly obtained from the one or more machine-learned models or can be a downstream result of a chain of processing operations that includes an output of the one or more machine- learned models.

[0137] At 806, example method 800 can include receiving an evaluation signal associated with the output. The evaluation signal can be obtained using a loss function. Various determinations of loss can be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, contrastive loss, or various other loss functions. The evaluation signal can be computed using known ground-truth labels (e.g., supervised learning), predicted or estimated labels (e.g., semi- or self-supervised learning), or without labels (e.g., unsupervised learning). The evaluation signal can be a reward (e.g., for reinforcement learning). The reward can be computed using a machine-learned reward model configured to generate rewards based on output(s) received. The reward can be computed using feedback data describing human feedback on the output(s).

[0138] At 808, example method 800 can include updating the machine-learned model using the evaluation signal. For example, values for parameters of the machine-learned model(s) can be learned, in some embodiments, using various training or learning techniques, such as, for example, backwards propagation. For example, the evaluation signal can be backpropagated from the output (or another source of the evaluation signal) through the machine-learned model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the evaluation signal with respect to the parameter value(s)). For example, system(s) containing one or more machine-learned models can be trained in an end-to-end manner. Gradient descent techniques can be used to iteratively update the parameters over anumber of training iterations. In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. Example method 800 can include implementing a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

[0139] In some implementations, example method 800 can be implemented for training a machine-learned model from an initialized state to a fully trained state (e.g., when the model exhibits a desired performance profile, such as based on accuracy, precision, recall, etc.).

[0140] In some implementations, example method 800 can be implemented for particular stages of a training procedure. For instance, in some implementations, example method 800 can be implemented for pre-training a machine-learned model. Pre-training can include, for instance, large-scale training over potentially noisy data to achieve a broad base of performance levels across a variety of tasks / data types. In some implementations, example method 800 can be implemented for fine-tuning a machine-learned model. Fine-tuning can include, for instance, smaller-scale training on higher-quality (e.g., labeled, curated, etc.) data. Fine-tuning can affect all or a portion of the parameters of a machine-learned model. For example, various portions of the machine-learned model can be “frozen” for certain training stages. For example, parameters associated with an embedding space can be “frozen” during fine-tuning (e.g., to retain information learned from a broader domain(s) than present in the fine-tuning dataset(s)). An example fine-tuning approach includes reinforcement learning. Reinforcement learning can be based on user feedback on model performance during use.Example Machine-Learned Models

[0141] Figure 9 is a block diagram of an example processing flow for using machine- learned model(s) 1 to process input(s) 2 to generate output(s) 3.

[0142] Machine-learned model(s) 1 can be or include one or multiple machine- learned models or model components. Example machine-learned models can include neural networks (e.g., deep neural networks). Example machine-learned models can include nonlinear models or linear models. Example machine-learned models can use other architectures in lieu of or in addition to neural networks. Example machine-learned models can include decision tree based models, support vector machines, hidden Markov models, Bayesian networks, linear regression models, k-means clustering models, etc.

[0143] Example neural networks can include feed-forward neural networks, recurrent neural networks (RNNs), including long short-term memory (LSTM) based recurrent neural networks, convolutional neural networks (CNNs), diffusion models, generative-adversarial networks, or other forms of neural networks. Example neural networks can be deep neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multiheaded self-attention models.

[0144] Machine-learned model(s) 1 can include a single or multiple instances of the same model configured to operate on data from input(s) 2. Machine-learned model(s) 1 can include an ensemble of different models that can cooperatively interact to process data from input(s) 2. For example, machine-learned model(s) 1 can employ a mixture-of-experts structure. See, e.g., Zhou et al., Mixture-of-Experts with Expert Choice Routing, ARXIV:2202.09368V2 (Oct. 14, 2022).

[0145] Input(s) 2 can generally include or otherwise represent various types of data. Input(s) 2 can include one type or many different types of data. Output(s) 3 can be data of the same type(s) or of different types of data as compared to input(s) 2. Output(s) 3 can include one type or many different types of data.

[0146] Example data types for input(s) 2 or output(s) 3 include natural language text data, software code data (e.g., source code, object code, machine code, or any other form of computer-readable instructions or programming languages), machine code data (e.g., binary code, assembly code, or other forms of machine-readable instructions that can be executed directly by a computer’s central processing unit), assembly code data (e.g., low-level programming languages that use symbolic representations of machine code instructions to program a processing unit), genetic data or other chemical or biochemical data, image data, audio data, audiovisual data, haptic data, biometric data, medical data, financial data, statistical data, geographical data, astronomical data, historical data, sensor data generally (e.g., digital or analog values, such as voltage or other absolute or relative level measurement values from a real or artificial input, such as from an audio sensor, light sensor, displacement sensor, etc.), and the like. Data can be raw or processed and can be in any format or schema.

[0147] In multimodal inputs 2 or outputs 3, example combinations of data types include image data and audio data, image data and natural language data, natural language data and software code data, image data and biometric data, sensor data and medical data, etc. It is to be understood that any combination of data types in an input 2 or an output 3 can be present.

[0148] An example input 2 can include one or multiple data types, such as the example data types noted above. An example output 3 can include one or multiple data types, such as the example data types noted above. The data type(s) of input 2 can be the same as or different from the data type(s) of output 3. It is to be understood that the example data types noted above are provided for illustrative purposes only. Data types contemplated within the scope of the present disclosure are not limited to those examples noted above.Example Machine-Learned Sequence Processing Models

[0149] Figure 10 is a block diagram of an example implementation of an example machine-learned model configured to process sequences of information. For instance, an example implementation of machine-learned model(s) 1 can include machine-learned sequence processing model(s) 4. An example system can pass input(s) 2 to sequence processing model(s) 4. Sequence processing model(s) 4 can include one or more machine- learned components. Sequence processing model(s) 4 can process the data from input(s) 2 to obtain an input sequence 5. Input sequence 5 can include one or more input elements 5-1, 5- 2, . . . , 5-A , etc. obtained from input(s) 2. Sequence processing model 4 can process input sequence 5 using prediction layer(s) 6 to generate an output sequence 7. Output sequence 7 can include one or more output elements 7-1, 7-2, . . . , 7 -A, etc. generated based on input sequence 5. The system can generate output(s) 3 based on output sequence 7.

[0150] Sequence processing model(s) 4 can include one or multiple machine-learned model components configured to ingest, generate, or otherwise reason over sequences of information. For example, some example sequence processing models in the text domain are referred to as “Large Language Models,” or LLMs. See, e.g., PaLM 2 Technical Report, GOOGLE, https: / / ai.google / static / documents / palm2techreport.pdf (n.d.). Other example sequence processing models can operate in other domains, such as image domains, see, e.g., Dosovitskiy et al., An Image is Worth 16x16 Words: transformers for Image Recognition at Scale, ARXIV:2010.11929v2 (Jun. 3, 2021), audio domains, see, e.g., Agostinelli et al., MusicLM: Generating Music From Text, ARXIV:2301.1 1325V1 (Jan. 26, 2023), biochemical domains, see, e.g., Jumper et al., Highly accurate protein structure prediction with AlphaFold, 596 Nature 583 (Aug. 26, 2021), by way of example. Sequence processing model(s) 4 can process one or multiple types of data simultaneously. Sequence processing model(s) 4 can include relatively large models (e.g., more parameters, computationally expensive, etc.), relatively small models (e.g., fewer parameters, computationally lightweight, etc.), or both.

[0151] In general, sequence processing model(s) 4 can obtain input sequence 5 using data from input(s) 2. For instance, input sequence 5 can include a representation of data from input(s) 2 in a format understood by sequence processing model(s) 4. One or more machine- learned components of sequence processing model(s) 4 can ingest the data from input(s) 2, parse the data into pieces compatible with the processing architectures of sequence processing model(s) 4 (e.g., via “tokenization”), and project the pieces into an input space associated with prediction layer(s) 6 (e.g., via “embedding”).

[0152] Sequence processing model(s) 4 can ingest the data from input(s) 2 and parse the data into a sequence of elements to obtain input sequence 5. For example, a portion of input data from input(s) 2 can be broken down into pieces that collectively represent the content of the portion of the input data. The pieces can provide the elements of the sequence.

[0153] Elements 5-1, 5-2, . . . , 5-M can represent, in some cases, building blocks for capturing or expressing meaningful information in a particular data domain. For instance, the elements can describe “atomic units” across one or more domains. For example, for textual input source(s), the elements can correspond to groups of one or more words or sub-word components, such as sets of one or more characters.

[0154] For example, elements 5-1, 5-2, . . . , 5-M can represent tokens obtained using a tokenizer. For instance, a tokenizer can process a given portion of an input source and output a series of tokens (e.g., corresponding to input elements 5-1, 5-2, . . . , 5-A ) that represent the portion of the input source. Various approaches to tokenization can be used. For instance, textual input source(s) can be tokenized using a byte-pair encoding (BPE) technique. See, e.g., Kudo et al., SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, PROCEEDINGS OF THE 2018 CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING (System Demonstrations), pages 66-71 (October 31-November 4, 2018), https: / / aclanthology.org / D18-2012.pdf. Image-based input source(s) can be tokenized by extracting and serializing patches from an image.

[0155] In general, arbitrary data types can be serialized and processed into input sequence 5. It is to be understood that element(s) 5-1, 5-2, . . . , 5-M depicted in Figure 10 can be the tokens or can be the embedded representations thereof.

[0156] Prediction layer(s) 6 can predict one or more output elements 7-1, 7-2, . . . , 7- N based on the input elements. Prediction layer(s) 6 can include one or more machine-learned model architectures, such as one or more layers of learned parameters that manipulate and transform the input(s) to extract higher-order meaning from, and relationships between, inputelement(s) 5-1, 5-2, . . . , 5-M. In this manner, for instance, example prediction layer(s) 6 can predict new output element(s) in view of the context provided by input sequence 5.

[0157] Prediction layer(s) 6 can evaluate associations between portions of input sequence 5 and a particular output element. These associations can inform a prediction of the likelihood that a particular output follows the input context. For example, consider the textual snippet, “The carpenter’s toolbox was small and heavy. It was full of .” Example prediction layer(s) 6 can identify that “It” refers back to “toolbox” by determining a relationship between the respective embeddings. Example prediction layer(s) 6 can also link “It” to the attributes of the toolbox, such as “small” and “heavy.” Based on these associations, prediction layer(s) 6 can, for instance, assign a higher probability to the word “nails” than to the word “sawdust.”

[0158] A transformer is an example architecture that can be used in prediction layer(s) 4. See, e.g., Vaswani et al., Atention Is All You Need, ARXIV: 1706.03762v7 (Aug. 2, 2023). A transformer is an example of a machine-learned model architecture that uses an attention mechanism to compute associations between items within a context window. The context window can include a sequence that contains input sequence 5 and potentially one or more output element(s) 7-1, 7-2, . . . , 7-N. A transformer block can include one or more attention layer(s) and one or more post-attention layer(s) (e.g., feedforward layer(s), such as a multi-layer perceptron).

[0159] Prediction layer(s) 6 can include other machine-learned model architectures in addition to or in lieu of transformer-based architectures. For example, recurrent neural networks (RNNs) and long short-term memory (LSTM) models can also be used, as well as convolutional neural networks (CNNs). In general, prediction layer(s) 6 can leverage various kinds of artificial neural networks that can understand or generate sequences of information.

[0160] Output sequence 7 can include or otherwise represent the same or different data types as input sequence 5. For instance, input sequence 5 can represent textual data, and output sequence 7 can represent textual data. Input sequence 5 can represent image, audio, or audiovisual data, and output sequence 7 can represent textual data (e.g., describing the image, audio, or audiovisual data). It is to be understood that prediction layer(s) 6, and any other interstitial model components of sequence processing model(s) 4, can be configured to receive a variety of data types in input sequence(s) 5 and output a variety of data types in output sequence(s) 7.

[0161] Output sequence 7 can have various relationships to input sequence 5. Output sequence 7 can be a continuation of input sequence 5. Output sequence 7 can becomplementary to input sequence 5. Output sequence 7 can translate, transform, augment, or otherwise modify input sequence 5. Output sequence 7 can answer, evaluate, confirm, or otherwise respond to input sequence 5. Output sequence 7 can implement (or describe instructions for implementing) an instruction provided via input sequence 5.

[0162] Output sequence 7 can be generated autoregressively. For instance, for some applications, an output of one or more prediction layer(s) 6 can be passed through one or more output layers (e.g., softmax layer) to obtain a probability distribution over an output vocabulary (e.g., a textual or symbolic vocabulary) conditioned on a set of input elements in a context window. In this manner, for instance, output sequence 7 can be autoregressively generated by sampling a likely next output element, adding that element to the context window, and re-generating the probability distribution based on the updated context window, and sampling a likely next output element, and so forth.

[0163] Output sequence 7 can also be generated non-autoregressively. For instance, multiple output elements of output sequence 7 can be predicted together without explicit sequential conditioning on each other. See, e.g., Saharia et al., Non-Autoregressive Machine Translation with Latent Alignments, ARXIV:2004.07437V3 (NOV. 16, 2020).

[0164] Output sequence 7 can include one or multiple portions or elements. In an example content generation configuration, output sequence 7 can include multiple elements corresponding to multiple portions of a generated output sequence (e.g., a textual sentence, values of a discretized waveform, computer code, etc.). In an example classification configuration, output sequence 7 can include a single element associated with a classification output. For instance, an output “vocabulary” can include a set of classes into which an input sequence is to be classified. For instance, a vision transformer block can pass latent state information to a multilayer perceptron that outputs a likely class value associated with an input image.

[0165] Figure 11 is a block diagram of an example technique for populating an example input sequence 8. Input sequence 8 can include various functional elements that form part of the model infrastructure, such as an element 8-0 obtained from a task indicator 9 that signals to any model(s) that process input sequence 8 that a particular task is being performed (e.g., to help adapt a performance of the model(s) to that particular task). Input sequence 8 can include various data elements from different data modalities. For instance, an input modality 10-1 can include one modality of data. A data-to- sequence model 11-1 can process data from input modality 10-1 to project the data into a format compatible with input sequence 8 (e.g., one or more vectors dimensioned according to the dimensions of inputsequence 8) to obtain elements 8-1, 8-2, 8-3. Another input modality 10-2 can include a different modality of data. A data-to-sequence model 11-2 can project data from input modality 10-2 into a format compatible with input sequence 8 to obtain elements 8-4, 8-5, 8- 6. Another input modality 10-3 can include yet another different modality of data. A data-to- sequence model 11-3 can project data from input modality 10-3 into a format compatible with input sequence 8 to obtain elements 8-7, 8-8, 8-9.

[0166] Input sequence 8 can be the same as or different from input sequence 5. Input sequence 8 can be a multimodal input sequence that contains elements that represent data from different modalities using a common dimensional representation. For instance, an embedding space can have P dimensions. Input sequence 8 can be configured to contain a plurality of elements that have / Jdimensions. In this manner, for instance, example implementations can facilitate information extraction and reasoning across diverse data modalities by projecting data into elements in the same embedding space for comparison, combination, or other computations therebetween.

[0167] For example, elements 8-0, . . . , 8-9 can indicate particular locations within a multidimensional embedding space. Some elements can map to a set of discrete locations in the embedding space. For instance, elements that correspond to discrete members of a predetermined vocabulary of tokens can map to discrete locations in the embedding space that are associated with those tokens. Other elements can be continuously distributed across the embedding space. For instance, some data types can be broken down into continuously defined portions (e.g., image patches) that can be described using continuously distributed locations within the embedding space.

[0168] In some implementations, the expressive power of the embedding space may not be limited to meanings associated with any particular set of tokens or other building blocks. For example, a continuous embedding space can encode a spectrum of high-order information. An individual piece of information (e.g., a token) can map to a particular point in that space: for instance, a token for the word “dog” can be projected to an embedded value that points to a particular location in the embedding space associated with canine-related information. Similarly, an image patch of an image of a dog on grass can also be projected into the embedding space. In some implementations, the projection of the image of the dog can be similar to the projection of the word “dog” while also having similarity to a projection of the word “grass,” while potentially being different from both. In some implementations, the projection of the image patch may not exactly align with any single projection of a single word. In some implementations, the projection of the image patch can align with acombination of the projections of the words “dog” and “grass.” In this manner, for instance, a high-order embedding space can encode information that can be independent of data modalities in which the information is expressed.

[0169] Task indicator 9 can include a model or model component configured to identify a task being performed and inject, into input sequence 8, an input value represented by element 8-0 that signals which task is being performed. For instance, the input value can be provided as a data type associated with an input modality and projected along with that input modality (e.g., the input value can be a textual task label that is embedded along with other textual data in the input; the input value can be a pixel-based representation of a task that is embedded along with other image data in the input; etc.). The input value can be provided as a data type that differs from or is at least independent from other input(s). For instance, the input value represented by element 8-0 can be a learned within a continuous embedding space.

[0170] Input modalities 10-1, 10-2, and 10-3 can be associated with various different data types (e.g., as described above with respect to input(s) 2 and output(s) 3).

[0171] Data-to-sequence models 11-1, 11-2, and 11-3 can be the same or different from each other. Data-to-sequence models 11-1, 11-2, and 11-3 can be adapted to each respective input modality 10-1, 10-2, and 10-3. For example, a textual data-to-sequence model can subdivide a portion of input text and project the subdivisions into element(s) in input sequence 8 (e.g., elements 8-1, 8-2, 8-3, etc.). An image data-to-sequence model can subdivide an input image and project the subdivisions into element(s) in input sequence 8 (e.g., elements 8-4, 8-5, 8-6, etc.). An arbitrary datatype data-to-sequence model can subdivide an input of that arbitrary datatype and project the subdivisions into element(s) in input sequence 8 (e.g., elements 8-7, 8-8, 8-9, etc.).

[0172] Data-to-sequence models 11-1, 11-2, and 11-3 can form part of machine- learned sequence processing model(s) 4. Data-to-sequence models 11-1, 11-2, and 11-3 can be jointly trained with or trained independently from machine-learned sequence processing model(s) 4. Data-to-sequence models 11-1, 11-2, and 11-3 can be trained end-to-end with machine-learned sequence processing model(s) 4.Example Machine-Learned Model Development Platform

[0173] Figure 12 is a block diagram of an example model development platform 12 that can facilitate creation, adaptation, and refinement of example machine-learned models (e.g., machine-learned model(s) 1, sequence processing model(s) 4, etc.). Model developmentplatform 12 can provide a number of different toolkits that developer systems can employ in the development of new or adapted machine-learned models.

[0174] Model development platform 12 can provide one or more model libraries 13 containing building blocks for new models. Model libraries 13 can include one or more pretrained foundational models 13-1, which can provide a backbone of processing power across various tasks. Model libraries 13 can include one or more pre-trained expert models 13-2, which can be focused on performance in particular domains of expertise. Model libraries 13 can include various model primitives 13-3, which can provide low-level architectures or components (optionally pre-trained), which can be assembled in various arrangements as desired.

[0175] Model development platform 12 can receive selections of various model components 14. Model development platform 12 can pass selected model components 14 to a workbench 15 that combines selected model components 14 into a development model 16.

[0176] Workbench 15 can facilitate further refinement and adaptation of development model 16 by leveraging a number of different toolkits integrated with model development platform 12. For example, workbench 15 can facilitate alignment of the development model 16 with a desired performance profile on various tasks using a model alignment toolkit 17.

[0177] Model alignment toolkit 17 can provide a number of tools for causing development model 16 to generate outputs aligned with desired behavioral characteristics. Alignment can include increasing an accuracy, precision, recall, etc. of model outputs.Alignment can include enforcing output styles, schema, or other preferential characteristics of model outputs. Alignment can be general or domain-specific. For instance, a pre-trained foundational model 13-1 can begin with an initial level of performance across multiple domains. Alignment of the pre-trained foundational model 13-1 can include improving a performance in a particular domain of information or tasks (e.g., even at the expense of performance in another domain of information or tasks).

[0178] Model alignment toolkit 17 can integrate one or more dataset(s) 17-1 for aligning development model 16. Curated dataset(s) 17-1 can include labeled or unlabeled training data. Dataset(s) 17-1 can be obtained from public domain datasets. Dataset(s) 17-1 can be obtained from private datasets associated with one or more developer system(s) for the alignment of bespoke machine-learned model(s) customized for private use-cases.

[0179] Pre-training pipelines 17-2 can include a machine-learned model training workflow configured to update development model 16 over large-scale, potentially noisy datasets. For example, pre-training can leverage unsupervised learning techniques (e.g., de-noising, etc.) to process large numbers of training instances to update model parameters from an initialized state and achieve a desired baseline performance. Pre-training pipelines 17-2 can leverage unlabeled datasets in dataset(s) 17-1 to perform pre-training. Workbench 15 can implement a pre-training pipeline 17-2 to pre-train development model 16.

[0180] Fine-tuning pipelines 17-3 can include a machine-learned model training workflow configured to refine the model parameters of development model 16 with higher- quality data. Fine-tuning pipelines 17-3 can update development model 16 by conducting supervised training with labeled dataset(s) in dataset(s) 17-1. Fine-tuning pipelines 17-3 can update development model 16 by conducting reinforcement learning using reward signals from user feedback signals. Workbench 15 can implement a fine-tuning pipeline 17-3 to finetune development model 16.

[0181] Prompt libraries 17-4 can include sets of inputs configured to induce behavior aligned with desired performance criteria. Prompt libraries 17-4 can include few-shot prompts (e.g., inputs providing examples of desired model outputs for prepending to a desired runtime query), chain-of-thought prompts (e.g., inputs providing step-by-step reasoning within the exemplars to facilitate thorough reasoning by the model), and the like.

[0182] Example prompts can be retrieved from an available repository of prompt libraries 17-4. Example prompts can be contributed by one or more developer systems using workbench 15.

[0183] In some implementations, pre-trained or fine-tuned models can achieve satisfactory performance without exemplars in the inputs. For instance, zero-shot prompts can include inputs that lack exemplars. Zero-shot prompts can be within a domain within a training dataset or outside of the training domain(s).

[0184] Prompt libraries 17-4 can include one or more prompt engineering tools. Prompt engineering tools can provide workflows for retrieving or learning optimized prompt values. Prompt engineering tools can facilitate directly learning prompt values (e.g., input element values) based one or more training iterations. Workbench 15 can implement prompt engineering tools in development model 16.

[0185] Prompt libraries 17-4 can include pipelines for prompt generation. For example, inputs can be generated using development model 16 itself or other machine- learned models. In this manner, for instance, a first model can process information about a task and output a input for a second model to process in order to perform a step of the task. The second model can be the same as or different from the first model. Workbench 15 can implement prompt generation pipelines in development model 16.

[0186] Prompt libraries 17-4 can include pipelines for context injection. For instance, a performance of development model 16 on a particular task can improve if provided with additional context for performing the task. Prompt libraries 17-4 can include software components configured to identify desired context, retrieve the context from an external source (e.g., a database, a sensor, etc.), and add the context to the input prompt. Workbench 15 can implement context injection pipelines in development model 16.

[0187] Although various training examples described herein with respect to model development platform 12 refer to “pre-training” and “fine-tuning,” it is to be understood that model alignment toolkit 17 can generally support a wide variety of training techniques adapted for training a wide variety of machine-learned models. Example training techniques can correspond to the example training method 800 described above.

[0188] Model development platform 12 can include a model plugin toolkit 18. Model plugin toolkit 18 can include a variety of tools configured for augmenting the functionality of a machine-learned model by integrating the machine-learned model with other systems, devices, and software components. For instance, a machine-learned model can use tools to increase performance quality where appropriate. For instance, deterministic tasks can be offloaded to dedicated tools in lieu of probabilistically performing the task with an increased risk of error. For instance, instead of autoregressively predicting the solution to a system of equations, a machine-learned model can recognize a tool to call for obtaining the solution and pass the system of equations to the appropriate tool. The tool can be a traditional system of equations solver that can operate deterministically to resolve the system of equations. The output of the tool can be returned in response to the original query. In this manner, tool use can allow some example models to focus on the strengths of machine-learned models — e.g., understanding an intent in an unstructured request for a task — while augmenting the performance of the model by offloading certain tasks to a more focused tool for rote application of deterministic algorithms to a well-defined problem.

[0189] Model plugin toolkit 18 can include validation tools 18-1. Validation tools 18- 1 can include tools that can parse and confirm output(s) of a machine-learned model. Validation tools 18-1 can include engineered heuristics that establish certain thresholds applied to model outputs. For example, validation tools 18-1 can ground the outputs of machine-learned models to structured data sources (e.g., to mitigate “hallucinations”).

[0190] Model plugin toolkit 18 can include tooling packages 18-2 for implementing one or more tools that can include scripts or other executable code that can be executed alongside development model 16. Tooling packages 18-2 can include one or more inputsconfigured to cause machine-learned model(s) to implement the tools (e.g., few-shot prompts that induce a model to output tool calls in the proper syntax, etc.). Tooling packages 18-2 can include, for instance, fine-tuning training data for training a model to use a tool.

[0191] Model plugin toolkit 18 can include interfaces for calling external application programming interfaces (APIs) 18-3. For instance, in addition to or in lieu of implementing tool calls or tool code directly with development model 16, development model 16 can be aligned to output instruction that initiate API calls to send or obtain data via external systems.

[0192] Model plugin toolkit 18 can integrate with prompt libraries 17-4 to build a catalog of available tools for use with development model 16. For instance, a model can receive, in an input, a catalog of available tools, and the model can generate an output that selects a tool from the available tools and initiates a tool call for using the tool.

[0193] Model development platform 12 can include a computational optimization toolkit 19 for optimizing a computational performance of development model 16. For instance, tools for model compression 19-1 can allow development model 16 to be reduced in size while maintaining a desired level of performance. For instance, model compression 19-1 can include quantization workflows, weight pruning and sparsification techniques, etc. Tools for hardware acceleration 19-2 can facilitate the configuration of the model storage and execution formats to operate optimally on different hardware resources. For instance, hardware acceleration 19-2 can include tools for optimally sharding models for distributed processing over multiple processing units for increased bandwidth, lower unified memory requirements, etc. Tools for distillation 19-3 can provide for the training of lighter-weight models based on the knowledge encoded in development model 16. For instance, development model 16 can be a highly performant, large machine-learned model optimized using model development platform 12. To obtain a lightweight model for running in resource-constrained environments, a smaller model can be a “student model” that learns to imitate development model 16 as a “teacher model.” In this manner, for instance, the investment in learning the parameters and configurations of development model 16 can be efficiently transferred to a smaller model for more efficient inference.

[0194] Workbench 15 can implement one, multiple, or none of the toolkits implemented in model development platform 12. Workbench 15 can output an output model 20 based on development model 16. Output model 20 can be a deployment version of development model 16. Output model 20 can be a development or training checkpoint of development model 16. Output model 20 can be a distilled, compressed, or otherwise optimized version of development model 16.

[0195] Figure 13 is a block diagram of an example training flow for training a machine-learned development model 16. One or more portion(s) of the example training flow can be implemented by a computing system that includes one or more computing devices such as, for example, computing systems described with reference to the other figures. Each respective portion of the example training flow can be performed by any (or any combination) of one or more computing devices. Moreover, one or more portion(s) of the example training flow can be implemented on the hardware components of the device(s) described herein, for example, to train one or more systems or models. FIG. 13 depicts elements performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, combined, or modified in various ways without deviating from the scope of the present disclosure. FIG. 13 is described with reference to elements / terms described with respect to other systems and figures for exemplary illustrated purposes and is not meant to be limiting. One or more portions of the example training flow can be performed additionally, or alternatively, by other systems.

[0196] Initially, development model 16 can persist in an initial state as an initialized model 21. Development model 16 can be initialized with weight values. Initial weight values can be random or based on an initialization schema. Initial weight values can be based on prior pre-training for the same or for a different model.

[0197] Initialized model 21 can undergo pre-training in a pre-training stage 22. Pretraining stage 22 can be implemented using one or more pre-training pipelines 17-2 over data from dataset(s) 17-1. Pre-training can be omitted, for example, if initialized model 21 is already pre-trained (e.g., development model 16 contains, is, or is based on a pre-trained foundational model or an expert model).

[0198] Pre-trained model 23 can then be a new version of development model 16, which can persist as development model 16 or as a new development model. Pre-trained model 23 can be the initial state if development model 16 was already pre-trained. Pre-trained model 23 can undergo fine-tuning in a fine-tuning stage 24. Fine-tuning stage 24 can be implemented using one or more fine-tuning pipelines 17-3 over data from dataset(s) 17-1. Fine-tuning can be omitted, for example, if a pre-trained model as satisfactory performance, if the model was already fine-tuned, or if other tuning approaches are preferred.

[0199] Fine-tuned model 29 can then be a new version of development model 16, which can persist as development model 16 or as a new development model. Fine-tunedmodel 29 can be the initial state if development model 16 was already fine-tuned. Fine-tuned model 29 can undergo refinement with user feedback 26. For instance, refinement with user feedback 26 can include reinforcement learning, optionally based on human feedback from human users of fine-tuned model 25. As reinforcement learning can be a form of fine-tuning, it is to be understood that fine-tuning stage 24 can subsume the stage for refining with user feedback 26. Refinement with user feedback 26 can produce a refined model 27. Refined model 27 can be output to downstream system(s) 28 for deployment or further development.

[0200] In some implementations, computational optimization operations can be applied before, during, or after each stage. For instance, initialized model 21 can undergo computational optimization 29-1 (e.g., using computational optimization toolkit 19) before pre-training stage 22. Pre-trained model 23 can undergo computational optimization 29-2 (e.g., using computational optimization toolkit 19) before fine-tuning stage 24. Fine-tuned model 25 can undergo computational optimization 29-3 (e.g., using computational optimization toolkit 19) before refinement with user feedback 26. Refined model 27 can undergo computational optimization 29-4 (e.g., using computational optimization toolkit 19) before output to downstream system(s) 28. Computational optimization(s) 29-1, . . . , 29-4 can all be the same, all be different, or include at least some different optimization techniques.Example Machine-Learned Model Inference System

[0201] Figure 14 is a block diagram of an inference system for operating one or more machine-learned model(s) 1 to perform inference (e.g., for training, for deployment, etc.). A model host 31 can receive machine-learned model(s) 1. Model host 31 can host one or more model instance(s) 31-1, which can be one or multiple instances of one or multiple models. Model host 31 can host model instance(s) 31-1 using available compute resources 31-2 associated with model host 31.

[0202] Model host 31 can perform inference on behalf of one or more client(s) 32. Client(s) 32 can transmit an input request 33 to model host 31. Using input request 33, model host 31 can obtain input(s) 2 for input to machine-learned model(s) 1. Machine-learned model(s) 1 can process input(s) 2 to generate output(s) 3. Using output(s) 3, model host 31 can return an output payload 34 for responding to input request 33 from client(s) 32. Output payload 34 can include or be based on output(s) 3.

[0203] Model host 31 can leverage various other resources and tools to augment the inference task. For instance, model host 31 can communicate with tool interfaces 35 tofacilitate tool use by model instance(s) 31-1. Tool interfaces 35 can include local or remote APIs. Tool interfaces 35 can include integrated scripts or other software functionality. Model host 31 can engage online learning interface(s) 36 to facilitate ongoing improvements to machine-learned model(s) 1. For instance, online learning interface(s) 36 can be used within reinforcement learning loops to retrieve user feedback on inferences served by model host 31. Model host 31 can access runtime data source(s) 37 for augmenting input(s) 2 with additional contextual information. For instance, runtime data source(s) 37 can include a knowledge graph 37-1 that facilitates structured information retrieval for information associated with input request(s) 33 (e.g., a search engine service). Runtime data source(s) 37 can include public or private, external or local database(s) 37-2 that can store information associated with input request(s) 33 for augmenting input(s) 2. Runtime data source(s) 37 can include account data 37-3 which can be retrieved in association with a user account corresponding to a client 32 for customizing the behavior of model host 31 accordingly.

[0204] Model host 31 can be implemented by one or multiple computing devices or systems. Client(s) 2 can be implemented by one or multiple computing devices or systems, which can include computing devices or systems shared with model host 31.

[0205] For example, model host 31 can operate on a server system that provides a machine-learning service to client device(s) that operate client(s) 32 (e.g., over a local or wide-area network). Client device(s) can be end-user devices used by individuals. Client device(s) can be server systems that operate client(s) 32 to provide various functionality as a service to downstream end-user devices.

[0206] In some implementations, model host 31 can operate on a same device or system as client(s) 32. Model host 31 can be a machine-learning service that runs on-device to provide machine-learning functionality to one or multiple applications operating on a client device, which can include an application implementing client(s) 32. Model host 31 can be a part of a same application as client(s) 32. For instance, model host 31 can be a subroutine or method implemented by one part of an application, and client(s) 32 can be another subroutine or method that engages model host 31 to perform inference functions within the application. It is to be understood that model host 31 and client(s) 32 can have various different configurations.

[0207] Model instance(s) 31-1 can include one or more machine-learned models that are available for performing inference. Model instance(s) 31-1 can include weights or other model components that are stored on in persistent storage, temporarily cached, or loaded into high-speed memory. Model instance(s) 31-1 can include multiple instance(s) of the samemodel (e.g., for parallel execution of more requests on the same model). Model instance(s) 31-1 can include instance(s) of different model(s). Model instance(s) 31-1 can include cached intermediate states of active or inactive model(s) used to accelerate inference of those models. For instance, an inference session with a particular model may generate significant amounts of computational results that can be re-used for future inference runs (e.g., using a KV cache for transformer-based models). These computational results can be saved in association with that inference session so that session can be executed more efficiently when resumed.

[0208] Compute resource(s) 31-2 can include one or more processors (central processing units, graphical processing units, tensor processing units, machine-learning accelerators, etc.) connected to one or more memory devices. Compute resource(s) 31-2 can include a dynamic pool of available resources shared with other processes. Compute resource(s) 31-2 can include memory devices large enough to fit an entire model instance in a single memory instance. Compute resource(s) 31-2 can also shard model instance(s) across multiple memory devices (e.g., using data parallelization or tensor parallelization, etc.). This can be done to increase parallelization or to execute a large model using multiple memory devices which individually might not be able to fit the entire model into memory.

[0209] Input request 33 can include data for input(s) 2. Model host 31 can process input request 33 to obtain input(s) 2. Input(s) 2 can be obtained directly from input request 33 or can be retrieved using input request 33. Input request 33 can be submitted to model host 31 via an API.

[0210] Model host 31 can perform inference over batches of input requests 33 in parallel. For instance, a model instance 31-1 can be configured with an input structure that has a batch dimension. Separate input(s) 2 can be distributed across the batch dimension (e.g., rows of an array). The separate input(s) 2 can include completely different contexts. The separate input(s) 2 can be multiple inference steps of the same task. The separate input(s) 2 can be staggered in an input structure, such that any given inference cycle can be operating on different portions of the respective input(s) 2. In this manner, for instance, model host 31 can perform inference on the batch in parallel, such that output(s) 3 can also contain the batch dimension and return the inference results for the batched input(s) 2 in parallel. In this manner, for instance, batches of input request(s) 33 can be processed in parallel for higher throughput of output payload(s) 34.

[0211] Output payload 34 can include or be based on output(s) 3 from machine- learned model(s) 1. Model host 31 can process output(s) 3 to obtain output payload 34. Thiscan include chaining multiple rounds of inference (e.g., iteratively, recursively, across the same model(s) or different model(s)) to arrive at a final output for a task to be returned in output payload 34. Output payload 34 can be transmitted to client(s) 32 via an API.

[0212] Online learning interface(s) 36 can facilitate reinforcement learning of machine-learned model(s) 1. Online learning interface(s) 36 can facilitate reinforcement learning with human feedback (RLHF). Online learning interface(s) 36 can facilitate federated learning of machine-learned model(s) 1.

[0213] Model host 31 can execute machine-learned model(s) 1 to perform inference for various tasks using various types of data. For example, various different input(s) 2 and output(s) 3 can be used for various different tasks. In some implementations, input(s) 2 can be or otherwise represent image data. Machine-learned model(s) 1 can process the image data to generate an output. As an example, machine-learned model(s) 1 can process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, machine-learned model(s) 1 can process the image data to generate an image segmentation output. As another example, machine-learned model(s) 1 can process the image data to generate an image classification output. As another example, machine-learned model(s) 1 can process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.). As another example, machine- learned model(s) 1 can process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, machine-learned model(s) 1 can process the image data to generate an upscaled image data output. As another example, machine-learned model(s) 1 can process the image data to generate a prediction output.

[0214] In some implementations, the task is a computer vision task. In some cases, input(s) 2 includes pixel data for one or more images and the task is an image processing task. For example, the image processing task can be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the likelihood that the one or more images depict an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in the one or more images and, for each region, a likelihood that region depicts an object of interest. As another example, the image processing task can be image segmentation, where the image processing output defines, for each pixel in the one or more images, a respective likelihood for each category in a predetermined set of categories. Forexample, the set of categories can be foreground and background. As another example, the set of categories can be object classes. As another example, the image processing task can be depth estimation, where the image processing output defines, for each pixel in the one or more images, a respective depth value. As another example, the image processing task can be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel of one of the input images, a motion of the scene depicted at the pixel between the images in the network input.

[0215] In some implementations, input(s) 2 can be or otherwise represent natural language data. Machine-learned model(s) 1 can process the natural language data to generate an output. As an example, machine-learned model(s) 1 can process the natural language data to generate a language encoding output. As another example, machine-learned model(s) 1 can process the natural language data to generate a latent text embedding output. As another example, machine-learned model(s) 1 can process the natural language data to generate a translation output. As another example, machine-learned model(s) 1 can process the natural language data to generate a classification output. As another example, machine-learned model(s) 1 can process the natural language data to generate a textual segmentation output. As another example, machine-learned model(s) 1 can process the natural language data to generate a semantic intent output. As another example, machine-learned model(s) 1 can process the natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is higher quality than the input text or natural language, etc.). As another example, machine-learned model(s) 1 can process the natural language data to generate a prediction output (e.g., one or more predicted next portions of natural language content).

[0216] In some implementations, input(s) 2 can be or otherwise represent speech data (e.g., data describing spoken natural language, such as audio data, textual data, etc.). Machine-learned model(s) 1 can process the speech data to generate an output. As an example, machine-learned model(s) 1 can process the speech data to generate a speech recognition output. As another example, machine-learned model(s) 1 can process the speech data to generate a speech translation output. As another example, machine-learned model(s) 1 can process the speech data to generate a latent embedding output. As another example, machine-learned model(s) 1 can process the speech data to generate an encoded speech output (e.g., an encoded and / or compressed representation of the speech data, etc.). As another example, machine-learned model(s) 1 can process the speech data to generate an upscaled speech output (e.g., speech data that is higher quality than the input speech data,etc.). As another example, machine-learned model(s) 1 can process the speech data to generate a textual representation output (e.g., a textual representation of the input speech data, etc.). As another example, machine-learned model(s) 1 can process the speech data to generate a prediction output.

[0217] In some implementations, input(s) 2 can be or otherwise represent latent encoding data (e.g., a latent space representation of an input, etc.). Machine-learned model(s) 1 can process the latent encoding data to generate an output. As an example, machine- learned model(s) 1 can process the latent encoding data to generate a recognition output. As another example, machine-learned model(s) 1 can process the latent encoding data to generate a reconstruction output. As another example, machine-learned model(s) 1 can process the latent encoding data to generate a search output. As another example, machine- learned model(s) 1 can process the latent encoding data to generate a reclustering output. As another example, machine-learned model(s) 1 can process the latent encoding data to generate a prediction output.

[0218] In some implementations, input(s) 2 can be or otherwise represent statistical data. Statistical data can be, represent, or otherwise include data computed and / or calculated from some other data source. Machine-learned model(s) 1 can process the statistical data to generate an output. As an example, machine-learned model(s) 1 can process the statistical data to generate a recognition output. As another example, machine-learned model(s) 1 can process the statistical data to generate a prediction output. As another example, machine- learned model(s) 1 can process the statistical data to generate a classification output. As another example, machine-learned model(s) 1 can process the statistical data to generate a segmentation output. As another example, machine-learned model(s) 1 can process the statistical data to generate a visualization output. As another example, machine-learned model(s) 1 can process the statistical data to generate a diagnostic output.

[0219] In some implementations, input(s) 2 can be or otherwise represent sensor data. Machine-learned model(s) 1 can process the sensor data to generate an output. As an example, machine-learned model(s) 1 can process the sensor data to generate a recognition output. As another example, machine-learned model(s) 1 can process the sensor data to generate a prediction output. As another example, machine-learned model(s) 1 can process the sensor data to generate a classification output. As another example, machine-learned model(s) 1 can process the sensor data to generate a segmentation output. As another example, machine-learned model(s) 1 can process the sensor data to generate a visualization output. As another example, machine-learned model(s) 1 can process the sensor data togenerate a diagnostic output. As another example, machine-learned model(s) 1 can process the sensor data to generate a detection output.

[0220] In some implementations, machine-learned model(s) 1 can be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task may be an audio compression task. The input may include audio data and the output may comprise compressed audio data. In another example, the input includes visual data (e.g. one or more images or videos), the output comprises compressed visual data, and the task is a visual data compression task. In another example, the task may comprise generating an embedding for input data (e.g. input audio or visual data). In some cases, the input includes audio data representing a spoken utterance and the task is a speech recognition task. The output may comprise a text output which is mapped to the spoken utterance. In some cases, the task comprises encrypting or decrypting input data. In some cases, the task comprises a microprocessor performance task, such as branch prediction or memory address translation.

[0221] In some implementations, the task is a generative task, and machine-learned model(s) 1 can be configured to output content generated in view of input(s) 2. For instance, input(s) 2 can be or otherwise represent data of one or more modalities that encodes context for generating additional content.

[0222] In some implementations, the task can be a text completion task. Machine- learned model(s) 1 can be configured to process input(s) 2 that represent textual data and to generate output(s) 3 that represent additional textual data that completes a textual sequence that includes input(s) 2. For instance, machine-learned model(s) 1 can be configured to generate output(s) 3 to complete a sentence, paragraph, or portion of text that follows from a portion of text represented by input(s) 2.

[0223] In some implementations, the task can be an instruction following task. Machine-learned model(s) 1 can be configured to process input(s) 2 that represent instructions to perform a function and to generate output(s) 3 that advance a goal of satisfying the instruction function (e.g., at least a step of a multi-step procedure to perform the function). Output(s) 3 can represent data of the same or of a different modality as input(s) 2. For instance, input(s) 2 can represent textual data (e.g., natural language instructions for a task to be performed) and machine-learned model(s) 1 can process input(s) 2 to generate output(s) 3 that represent textual data responsive to the instructions (e.g., natural language responses, programming language responses, machine language responses, etc.). Input(s) 2 can represent image data (e.g., image-based instructions for a task to be performed, optionallyaccompanied by textual instructions) and machine-learned model(s) 1 can process input(s) 2 to generate output(s) 3 that represent textual data responsive to the instructions (e.g., natural language responses, programming language responses, machine language responses, etc.). One or more output(s) 3 can be iteratively or recursively generated to sequentially process and accomplish steps toward accomplishing the requested functionality. For instance, an initial output can be executed by an external system or be processed by machine-learned model(s) 1 to complete an initial step of performing a function. Multiple steps can be performed, with a final output being obtained that is responsive to the initial instructions.

[0224] In some implementations, the task can be a question answering task. Machine- learned model(s) 1 can be configured to process input(s) 2 that represent a question to answer and to generate output(s) 3 that advance a goal of returning an answer to the question (e.g., at least a step of a multi-step procedure to perform the function). Output(s) 3 can represent data of the same or of a different modality as input(s) 2. For instance, input(s) 2 can represent textual data (e.g., natural language instructions for a task to be performed) and machine- learned model(s) 1 can process input(s) 2 to generate output(s) 3 that represent textual data responsive to the question (e.g., natural language responses, programming language responses, machine language responses, etc.). Input(s) 2 can represent image data (e.g., image-based instructions for a task to be performed, optionally accompanied by textual instructions) and machine-learned model(s) 1 can process input(s) 2 to generate output(s) 3 that represent textual data responsive to the question (e.g., natural language responses, programming language responses, machine language responses, etc.). One or more output(s) 3 can be iteratively or recursively generated to sequentially process and accomplish steps toward answering the question. For instance, an initial output can be executed by an external system or be processed by machine-learned model(s) 1 to complete an initial step of obtaining an answer to the question (e.g., querying a database, performing a computation, executing a script, etc.). Multiple steps can be performed, with a final output being obtained that is responsive to the question.

[0225] In some implementations, the task can be an image generation task. Machine- learned model(s) 1 can be configured to process input(s) 2 that represent context regarding a desired portion of image content. The context can include text data, image data, audio data, etc. Machine-learned model(s) 1 can be configured to generate output(s) 3 that represent image data that depicts imagery related to the context. For instance, machine-learned model(s) 1 can be configured to generate pixel data of an image. Values for channel(s)associated with the pixels in the pixel data can be selected based on the context (e.g., based on a probability determined based on the context).

[0226] In some implementations, the task can be an audio generation task. Machine- learned model(s) 1 can be configured to process input(s) 2 that represent context regarding a desired portion of audio content. The context can include text data, image data, audio data, etc. Machine-learned model(s) 1 can be configured to generate output(s) 3 that represent audio data related to the context. For instance, machine-learned model(s) 1 can be configured to generate waveform data in the form of an image (e.g., a spectrogram). Values for channel(s) associated with pixels of the image can be selected based on the context. Machine- learned model(s) 1 can be configured to generate waveform data in the form of a sequence of discrete samples of a continuous waveform. Values of the sequence can be selected based on the context (e.g., based on a probability determined based on the context).

[0227] In some implementations, the task can be a data generation task. Machine- learned model(s) 1 can be configured to process input(s) 2 that represent context regarding a desired portion of data (e.g., data from various data domains, such as sensor data, image data, multimodal data, statistical data, etc.). The desired data can be, for instance, synthetic data for training other machine-learned models. The context can include arbitrary data type(s). Machine-learned model(s) 1 can be configured to generate output(s) 3 that represent data that aligns with the desired data. For instance, machine-learned model(s) 1 can be configured to generate data values for populating a dataset. Values for the data object(s) can be selected based on the context (e.g., based on a probability determined based on the context).Example Computing Systems and Devices

[0228] Figure 15 is a block diagram of an example networked computing system that can perform aspects of example implementations of the present disclosure. The system can include a number of computing devices and systems that are communicatively coupled over a network 49. An example computing device 50 is described to provide an example of a computing device that can perform any aspect of the present disclosure (e.g., implementing model host 31, client(s) 32, or both). An example server computing system 60 is described as an example of a server computing system that can perform any aspect of the present disclosure (e.g., implementing model host 31, client(s) 32, or both). Computing device 50 and server computing system(s) 60 can cooperatively interact (e.g., over network 49) to perform any aspect of the present disclosure (e.g., implementing model host 31, client(s) 32, or both). Model development platform system 70 is an example system that can host or serve modeldevelopment platform(s) 12 for development of machine-learned models. Third-party system(s) 80 are example system(s) with which any of computing device 50, server computing system(s) 60, or model development platform system(s) 70 can interact in the performance of various aspects of the present disclosure (e.g., engaging third-party tools, accessing third-party databases or other resources, etc.).

[0229] Network 49 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links. In general, communication over network 49 can be carried via any type of wired or wireless connection, using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), or protection schemes (e.g., VPN, secure HTTP, SSL). Network 49 can also be implemented via a system bus. For instance, one or more devices or systems of Figure 15 can be co-located with, contained by, or otherwise integrated into one or more other devices or systems.

[0230] Computing device 50 can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, a server computing device, a virtual machine operating on a host device, or any other type of computing device. Computing device 50 can be a client computing device. Computing device 50 can be an end-user computing device. Computing device 50 can be a computing device of a service provided that provides a service to an end user (who may use another computing device to interact with computing device 50).

[0231] Computing device 50 can include one or more processors 51 and a memory 52. Processor(s) 51 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. Memory 52 can include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. Memory 52 can store data 53 and instructions 54 which can be executed by processor(s) 51 to cause computing device 50 to perform operations. The operations can implement any one or multiple features described herein. The operations can implement example methods and techniques described herein.

[0232] Computing device 50 can also include one or more input components that receive user input. For example, a user input component can be a touch-sensitive component(e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, camera, LIDAR, a physical keyboard or other buttons, or other means by which a user can provide user input.

[0233] Computing device 50 can store or include one or more machine-learned models 55. Machine-learned models 55 can include one or more machine-learned model(s) 1, such as a sequence processing model 4. Machine-learned models 55 can include one or multiple model instance(s) 31-1. Machine-learned model(s) 55 can be received from server computing system(s) 60, model development platform system 70, third party system(s) 80 (e.g., an application distribution platform), or developed locally on computing device 50. Machine-learned model(s) 55 can be loaded into memory 52 and used or otherwise implemented by processor(s) 51. Computing device 50 can implement multiple parallel instances of machine-learned model(s) 55.

[0234] Server computing system(s) 60 can include one or more processors 61 and a memory 62. Processor(s) 61 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. Memory 62 can include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. Memory 62 can store data 63 and instructions 64 which can be executed by processor(s) 61 to cause server computing system(s) 60 to perform operations. The operations can implement any one or multiple features described herein. The operations can implement example methods and techniques described herein.

[0235] In some implementations, server computing system 60 includes or is otherwise implemented by one or multiple server computing devices. In instances in which server computing system 60 includes multiple server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

[0236] Server computing system 60 can store or otherwise include one or more machine-learned models 65. Machine-learned model(s) 65 can be the same as or different from machine-learned model(s) 55. Machine-learned models 65 can include one or more machine-learned model(s) 1, such as a sequence processing model 4. Machine-learned models 65 can include one or multiple model instance(s) 31-1. Machine-learned model(s) 65can be received from computing device 50, model development platform system 70, third party system(s) 80, or developed locally on server computing system(s) 60. Machine-learned model(s) 65 can be loaded into memory 62 and used or otherwise implemented by processor(s) 61. Server computing system(s) 60 can implement multiple parallel instances of machine-learned model (s) 65.

[0237] In an example configuration, machine-learned models 65 can be included in or otherwise stored and implemented by server computing system 60 to establish a client-server relationship with computing device 50 for serving model inferences. For instance, server computing system(s) 60 can implement model host 31 on behalf of client(s) 32 on computing device 50. For instance, machine-learned models 65 can be implemented by server computing system 60 as a portion of a web service (e.g., remote machine-learned model hosting service, such as an online interface for performing machine-learned model operations over a network on server computing system(s) 60). For instance, server computing system(s) 60 can communicate with computing device 50 over a local intranet or internet connection. For instance, computing device 50 can be a workstation or endpoint in communication with server computing system(s) 60, with implementation of machine-learned models 65 being managed by server computing system(s) 60 to remotely perform inference (e.g., for runtime or training operations), with output(s) returned (e.g., cast, streamed, etc.) to computing device 50. Machine-learned models 65 can work cooperatively or interoperatively with machine- learned models 55 on computing device 50 to perform various tasks.

[0238] Model development platform system(s) 70 can include one or more processors 71 and a memory 72. Processor(s) 71 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. Memory 72 can include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. Memory 72 can store data 73 and instructions 74 which can be executed by processor(s) 71 to cause model development platform system(s) 70 to perform operations. The operations can implement any one or multiple features described herein. The operations can implement example methods and techniques described herein. Example operations include the functionality described herein with respect to model development platform 12. This and other functionality can be implemented by developer tool(s) 75.

[0239] Third-party system(s) 80 can include one or more processors 81 and a memory 82. Processor(s) 81 can be any suitable processing device (e.g., a processor core, amicroprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. Memory 82 can include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. Memory 82 can store data 83 and instructions 84 which can be executed by processor(s) 81 to cause third-party system(s) 80 to perform operations. The operations can implement any one or multiple features described herein. The operations can implement example methods and techniques described herein. Example operations include the functionality described herein with respect to tools and other external resources called when training or performing inference with machine-learned model(s) 1, 4, 16, 20, 55, 65, etc. (e.g., third-party resource(s) 85).

[0240] Figure 15illustrates one example arrangement of computing systems that can be used to implement the present disclosure. Other computing system configurations can be used as well. For example, in some implementations, one or both of computing system 50 or server computing system(s) 60 can implement all or a portion of the operations of model development platform system 70. For example, computing system 50 or server computing system(s) 60 can implement developer tool(s) 75 (or extensions thereof) to develop, update / train, or refine machine-learned models 1, 4, 16, 20, 55, 65, etc. using one or more techniques described herein with respect to model alignment toolkit 17. In this manner, for instance, computing system 50 or server computing system(s) 60 can develop, update / train, or refine machine-learned models based on local datasets (e.g., for model personalization / customization, as permitted by user data preference selections).

[0241] Figure 16 is a block diagram of an example computing device 98 that performs according to example embodiments of the present disclosure. Computing device 98 can be a user computing device or a server computing device (e.g., computing device 50, server computing system(s) 60, etc.). Computing device 98 can implement model host 31. For instance, computing device 98 can include a number of applications (e.g., applications 1 through N). Each application can contain its own machine learning library and machine- learned model(s). For example, each application can include a machine-learned model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. As illustrated in Figure 16, each application can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, or additional components. In some implementations, each application cancommunicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0242] Figure 17 is a block diagram of an example computing device 99 that performs according to example embodiments of the present disclosure. Computing device 99 can be the same as or different from computing device 98. Computing device 99 can be a user computing device or a server computing device (e.g., computing device 50, server computing system(s) 60, etc.). Computing device 98 can implement model host 31. For instance, computing device 99 can include a number of applications (e.g., applications 1 through N). Each application can be in communication with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).

[0243] The central intelligence layer can include a number of machine-learned models. For example, as illustrated in Figure 17, a respective machine-learned model can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of computing device 99.

[0244] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for computing device 99. As illustrated in Figure 17, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).Additional Disclosure

[0245] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussedherein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0246] While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure cover such alterations, variations, and equivalents.

[0247] Aspects of the disclosure have been described in terms of illustrative embodiments thereof. Any and all features in the following claims can be combined or rearranged in any way possible, including combinations of claims not explicitly enumerated in combination together, as the example claim dependencies listed herein should not be read as limiting the scope of possible combinations of features disclosed herein. Accordingly, the scope of the present disclosure is by way of example rather than by way of limitation, and the subject disclosure does not preclude inclusion of such modifications, variations or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. Moreover, terms are described herein using lists of example elements joined by conjunctions such as “and,” “or,” “but,” etc. It should be understood that such conjunctions are provided for explanatory purposes only. Clauses and other sequences of items joined by a particular conjunction such as “or,” for example, can refer to “and / or,” “at least one of’, “any combination of’ example elements listed therein, etc. Terms such as “based on” should be understood as “based at least in part on.”

[0248] The term “can” should be understood as referring to a possibility of a feature in various implementations and not as prescribing an ability that is necessarily present in every implementation. For example, the phrase “X can perform Y” should be understood as indicating that, in various implementations, X has the potential to be configured to perform Y, and not as indicating that in every instance X must always be able to perform Y. It shouldbe understood that, in various implementations, X might be unable to perform Y and remain within the scope of the present disclosure.

[0249] The term “may” should be understood as referring to a possibility of a feature in various implementations and not as prescribing an ability that is necessarily present in every implementation. For example, the phrase “X may perform Y” should be understood as indicating that, in various implementations, X has the potential to be configured to perform Y, and not as indicating that in every instance X must always be able to perform Y. It should be understood that, in various implementations, X might be unable to perform Y and remain within the scope of the present disclosure.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method to perform self-training of sequence processing models to leverage external sources of knowledge, the method comprising: performing, by a computing system comprising one or more computing devices, a plurality of self-training iterations, each self-training iteration comprising: obtaining, by the computing system, an exemplar tuple comprising an exemplar query and an exemplar response, wherein the exemplar response comprises an exemplar chain of analysis that arrives at an exemplar answer to the exemplar query, and wherein the exemplar chain of analysis relies upon or makes reference to one or more exemplar items of content obtained from one or more external sources of knowledge; generating, by the computing system, a training prompt comprising the exemplar tuple, a training query, and one or more training items of content obtained from the one or more external sources of knowledge; processing, by the computing system, the training prompt with a sequence processing model to generate a candidate response as an output of the sequence processing model, wherein the candidate response comprises a model -generated chain of analysis that arrives at a candidate answer to the training query; determining, by the computing system, whether the candidate response corresponds to a ground truth answer to the training query; and in response to determining that the candidate response corresponds to the ground truth answer, fine-tuning the sequence processing model on the candidate response.

2. The computer-implemented method of claim 1, wherein the model-generated chain of analysis relies upon or makes reference to the one or more training items of content.

3. The computer-implemented method of any preceding claim, wherein generating, by the computing system, the training prompt comprises: processing, by the computing system, at least the training query with the sequence processing model or a different model to generate an information retrieval query; and retrieving, by the computing system, the one or more training items of content from a data repository based on the information retrieval query.

4. The computer-implemented method of any preceding claim, wherein the one or more external sources of knowledge comprise a search engine and the Internet, and wherein the one or more training items of content comprise web documents retrieved from the Internet using the search engine.

5. The computer-implemented method of any preceding claim, wherein the one or more external sources of knowledge comprise a database of medical knowledge or records, and wherein the one or more training items of content comprise medical articles or records retrieved from the database of medical knowledge or records.

6. The computer-implemented method of any preceding claim, wherein the training query comprises a medical question, and wherein the ground truth answer comprises a medical answer to the medical question.

7. The computer-implemented method of any preceding claim, further comprising: in response to determining that the candidate response corresponds to the ground truth answer, adding the training query and the candidate response to a set of exemplars from which the exemplar tuple was obtained.

8. The computer-implemented method of any preceding claim, further comprising: in response to determining that the candidate response does not correspond to the ground truth answer, discarding the candidate response.

9. The computer-implemented method of any preceding claim, wherein the exemplar response of the exemplar tuple comprises manually generated or edited content.

10. The computer-implemented method of any preceding claim, wherein each self-training iteration further comprises: obtaining, by the computing system, a second exemplar tuple comprising a second exemplar query and a second exemplar response, wherein the second exemplar response comprises a second exemplar chain of analysis that arrives at a second exemplar answer tothe exemplar query without reference to any items of content obtained from one or more external sources of knowledge; generating, by the computing system, a second training prompt comprising the second exemplar tuple and a second training query; processing, by the computing system, the second training prompt with a sequence processing model to generate a second candidate response as an output of the sequence processing model, wherein the candidate response comprises a second model-generated chain of analysis that arrives at a second candidate answer to the second training query without reference to any items of content obtained from one or more external sources of knowledge; determining, by the computing system, whether the second candidate response corresponds to a second ground truth answer to the second training query; and in response to determining that the second candidate response corresponds to the second ground truth answer, fine-tuning the sequence processing model on the second candidate response.

11. A computing system, comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising: obtaining, by the computing system, an exemplar tuple comprising an exemplar query and an exemplar response, wherein the exemplar response comprises an exemplar chain of analysis that arrives at an exemplar answer to the exemplar query, and wherein the exemplar chain of analysis relies upon or makes reference to one or more exemplar items of content obtained from one or more external sources of knowledge; generating, by the computing system, a training prompt comprising the exemplar tuple, a training query, and one or more training items of content obtained from the one or more external sources of knowledge; processing, by the computing system, the training prompt with a sequence processing model to generate a candidate response as an output of the sequence processing model, wherein the candidate response comprises a model -generated chain of analysis that arrives at a candidate answer to the training query; determining, by the computing system, whether the candidate response corresponds to a ground truth answer to the training query; andin response to determining that the candidate response corresponds to the ground truth answer, fine-tuning the sequence processing model on the candidate response.

12. The computing system of claim 11, wherein the model -generated chain of analysis relies upon or makes reference to the one or more training items of content.

13. The computing system of claim 11 or 12, wherein generating, by the computing system, the training prompt comprises: processing, by the computing system, at least the training query with the sequence processing model or a different model to generate an information retrieval query; and retrieving, by the computing system, the one or more training items of content from a data repository based on the information retrieval query.

14. The computing system of claim 11, 12, or 13, wherein the one or more external sources of knowledge comprise a search engine and the Internet, and wherein the one or more training items of content comprise web documents retrieved from the Internet using the search engine.

15. The computing system of any of claims 11-14, wherein the one or more external sources of knowledge comprise a database of medical knowledge or records, and wherein the one or more training items of content comprise medical articles or records retrieved from the database of medical knowledge or records.

16. The computing system of any of claims 11-15, wherein the training query comprises a medical question, and wherein the ground truth answer comprises a medical answer to the medical question.

17. The computing system of any of claims 11-16, further comprising: in response to determining that the candidate response corresponds to the ground truth answer, adding the training query and the candidate response to a set of exemplars from which the exemplar tuple was obtained.

18. The computing system of any of claims 11-17, further comprising:in response to determining that the candidate response does not correspond to the ground truth answer, discarding the candidate response.

19. The computing system of any of claims 11-18, wherein the exemplar response of the exemplar tuple comprises manually generated or edited content.

20. One or more non-transitory computer-readable media that collectively store a sequence processing model that has been trained by the performance of a plurality of selftraining iterations, each self-training iteration having included: obtaining, by a computing system, an exemplar tuple comprising an exemplar query and an exemplar response, wherein the exemplar response comprises an exemplar chain of analysis that arrives at an exemplar answer to the exemplar query, and wherein the exemplar chain of analysis relies upon or makes reference to one or more exemplar items of content obtained from one or more external sources of knowledge; generating, by the computing system, a training prompt comprising the exemplar tuple, a training query, and one or more training items of content obtained from the one or more external sources of knowledge; processing, by the computing system, the training prompt with the sequence processing model to generate a candidate response as an output of the sequence processing model, wherein the candidate response comprises a model -generated chain of analysis that arrives at a candidate answer to the training query; determining, by the computing system, whether the candidate response corresponds to a ground truth answer to the training query; and in response to determining that the candidate response corresponds to the ground truth answer, fine-tuning the sequence processing model on the candidate response.

21. A computer-implemented method for performing uncertainty-guided use of external sources of knowledge for response generation, the method comprising: obtaining, by a computing system comprising one or more computing devices, an input query; processing, by the computing system, the input query with a sequence processing model to generate a plurality of different responses to the input query as respective outputs of the sequence processing model;determining, by the computing system, whether the plurality of different responses satisfy a consensus metric; and in response to determining that the plurality of different responses do not satisfy the consensus metric: retrieving, by the computing system, one or more additional items of content from one or more external sources of knowledge; and processing, by the computing system, the input query and the one or more additional items of content with the sequence processing model to generate at least one additional response to the input query as an additional output of the sequence processing model.

22. The computer-implemented method of claim 21, wherein: processing, by the computing system, the input query and the one or more additional items of content with the sequence processing model to generate the at least one additional response comprises processing, by the computing system, the input query and the one or more additional items of content with the sequence processing model to generate a plurality of additional responses; and the method further comprises: determining, by the computing system, whether the plurality of additional responses satisfy the consensus metric; and in response to determining that the plurality of additional responses satisfy the consensus metric: providing at least a portion of at least one of the plurality of additional responses as an output.

23. The computer-implemented method of claim 22, wherein providing at least a portion of at least one of the plurality of additional responses as the output comprises providing a consensus answer from the plurality of additional responses as an output.

24. The computer-implemented method of any of claims 21-23, wherein retrieving, by the computing system, the one or more additional items of content from the one or more external sources of knowledge in response to determining that the plurality of different responses do not satisfy the consensus metric comprises: identifying, by the computing system, two or more of the plurality of different responses that exhibit a conflict;generating, by the computing system, an information retrieval query that returns items of content that relate to the conflict; and querying, by the computing system, the one or more external sources of knowledge with the information retrieval query to obtain the items of content.

25. The computer-implemented method of any claim 24, wherein generating, by the computing system, the information retrieval query that returns items of content that relate to the conflict comprises prompting the sequence processing model or a different model with the two or more of the plurality of different responses that exhibit the conflict and instructions to generate the information retrieval query.

26. The computer-implemented method of any of claims 21-25, wherein the one or more external sources of knowledge comprise a search engine and the Internet, and wherein the one or more items of content comprise web documents retrieved from the Internet using the search engine.

27. The computer-implemented method of any of claims 21-26, wherein the one or more external sources of knowledge comprise a database of medical knowledge or records, and wherein the one or more items of content comprise medical articles or records retrieved from the database of medical knowledge or records.

28. The computer-implemented method of any of claims 21-27, wherein the input query comprises a medical question, and wherein each of the plurality of different responses and the at least one additional response comprise a respective medical answer to the medical question.

29. The computer-implemented method of any of claims 21-28, wherein each of the plurality of different responses and the at least one additional response comprise a respective chain of analysis that leads from the input query to a respective answer.

30. The computer-implemented method of any of claims 21-29, wherein determining, by the computing system, whether the plurality of different responses satisfy the consensus metric comprises:determining, by the computing system, an entropy of a distribution of respective answers within the plurality of different responses; and comparing, by the computing system, the entropy of the distribution to a threshold value.

31. The computer-implemented method of any of claims 21-29, wherein determining, by the computing system, whether the plurality of different responses satisfy the consensus metric comprises prompting the sequence processing model or a different model with the plurality of different responses and a request to analyze whether the plurality of different responses exhibit consensus.

32. A computer system performing uncertainty-guided use of external sources of knowledge for response generation, the computer system comprising one or more processors and one or more non-transitory computer-readable media that store instructions executable by the processors to perform operations, the operations comprising: obtaining, by the computing system, an input query; processing, by the computing system, the input query with a sequence processing model to generate a plurality of different responses to the input query as respective outputs of the sequence processing model; determining, by the computing system, whether the plurality of different responses satisfy a consensus metric; and in response to determining that the plurality of different responses do not satisfy the consensus metric: retrieving, by the computing system, one or more additional items of content from one or more external sources of knowledge; and processing, by the computing system, the input query and the one or more additional items of content with the sequence processing model to generate at least one additional response to the input query as an additional output of the sequence processing model.

33. The computer system of claim 32, wherein: processing, by the computing system, the input query and the one or more additional items of content with the sequence processing model to generate the at least one additional response comprises processing, by the computing system, the input query and the one ormore additional items of content with the sequence processing model to generate a plurality of additional responses; and the operations further comprise: determining, by the computing system, whether the plurality of additional responses satisfy the consensus metric; and in response to determining that the plurality of additional responses satisfy the consensus metric: providing at least a portion of at least one of the plurality of additional responses as an output.

34. The computer system of claim 33, wherein providing at least a portion of at least one of the plurality of additional responses as the output comprises providing a consensus answer from the plurality of additional responses as an output.

35. The computer system of any of claims 32-34, wherein retrieving, by the computing system, the one or more additional items of content from the one or more external sources of knowledge in response to determining that the plurality of different responses do not satisfy the consensus metric comprises: identifying, by the computing system, two or more of the plurality of different responses that exhibit a conflict; generating, by the computing system, an information retrieval query that returns items of content that relate to the conflict; and querying, by the computing system, the one or more external sources of knowledge with the information retrieval query to obtain the items of content.

36. The computer system of claim 35, wherein generating, by the computing system, the information retrieval query that returns items of content that relate to the conflict comprises prompting the sequence processing model or a different model with the two or more of the plurality of different responses that exhibit the conflict and instructions to generate the information retrieval query.

37. The computer system of any of claims 32-36, wherein the one or more external sources of knowledge comprise a search engine and the Internet, and wherein the one or more items of content comprise web documents retrieved from the Internet using the search engine.

38. The computer system of any of claims 32-37, wherein determining, by the computing system, whether the plurality of different responses satisfy the consensus metric comprises: determining, by the computing system, an entropy of a distribution of respective answers within the plurality of different responses; and comparing, by the computing system, the entropy of the distribution to a threshold value.

39. The computer system of any of claims 32-37, wherein determining, by the computing system, whether the plurality of different responses satisfy the consensus metric comprises prompting the sequence processing model or a different model with the plurality of different responses and a request to analyze whether the plurality of different responses exhibit consensus.

40. One or more non-transitory computer-readable media that collectively store computer-executable instructions for performing operations, the operations comprising: obtaining, by a computing system, an input query; processing, by the computing system, the input query with a sequence processing model to generate a plurality of different responses to the input query as respective outputs of the sequence processing model; determining, by the computing system, whether the plurality of different responses satisfy a consensus metric; and in response to determining that the plurality of different responses do not satisfy the consensus metric: retrieving, by the computing system, one or more additional items of content from one or more external sources of knowledge; and processing, by the computing system, the input query and the one or more additional items of content with the sequence processing model to generate at least one additional response to the input query as an additional output of the sequence processing model.

41. A computer-implemented method to facilitate processing of large medical context inputs with improved accuracy, the method comprising:obtaining, by a computing system comprising one or more computing devices, an input medical query and a set of medical context data; performing, by the computing system, a multi-stage chain-of-prompting approach to process the input medical query and the set of medical context data with a sequence processing model to generate an output response as an output of the sequence processing model; and providing, by the computing system, the output response in response to the input medical query; wherein performing, by the computing system, the multi-stage chain-of-prompting approach comprises iteratively prompting, by the computing system, the sequence processing model with a series of prompts that are at least in part pre-defined; and wherein the series of prompts elicits, over the series of prompts, the sequence processing model to perform a chain of analysis on the input query and the set of medical context data to generate the output response.

42. The computer-implemented method of claim 41, wherein: the series of prompts comprises: at least a first prompt that instruct the sequence processing model to perform information retrieval from the set of medical context data to generate a smaller set of retrieved data; and at least a second prompt that instructs the sequence processing model to perform medical analysis on the smaller set of retrieved data; and the second prompt is subsequent to the first prompt within the series of prompts.

43. The computer-implemented method of claim 42, wherein: the input medical query relates to a particular disease, condition, or procedure; and the first prompt instructs the sequence processing model to retrieve all mentions of the particular disease, condition, or procedure within the set of medical context data.

44. The computer-implemented method of claim 43, wherein the first prompt instructs the sequence processing model to provide direct quotations from the set of medical context data.

45. The computer-implemented method of any of claims 41-44, wherein the set of medical context data comprises data associated with at least two different data modalities.

46. The computer-implemented method of any of claims 41-45, wherein the set of medical context data comprises both image data and text data.

47. The computer-implemented method of any of claims 41-46, wherein the set of medical context data comprises a plurality of electronic health records.

48. The computer-implemented method of any of claims 41-47, wherein the input medical query comprises a diagnostic query, and wherein the output response comprises an output diagnosis.

49. A computing system that processes large medical context inputs with improved accuracy, the computer system comprising one or more processors and one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising: obtaining, by the computing system, an input medical query and a set of medical context data; performing, by the computing system, a multi-stage chain-of-prompting approach to process the input medical query and the set of medical context data with a sequence processing model to generate an output response as an output of the sequence processing model; and providing, by the computing system, the output response in response to the input medical query; wherein performing, by the computing system, the multi-stage chain-of-prompting approach comprises iteratively prompting, by the computing system, the sequence processing model with a series of prompts that are at least in part pre-defined; and wherein the series of prompts elicits, over the series of prompts, the sequence processing model to perform a chain of analysis on the input query and the set of medical context data to generate the output response.

50. The computing system of claim 49, wherein:the series of prompts comprises: at least a first prompt that instruct the sequence processing model to perform information retrieval from the set of medical context data to generate a smaller set of retrieved data; and at least a second prompt that instructs the sequence processing model to perform medical analysis on the smaller set of retrieved data; and the second prompt is subsequent to the first prompt within the series of prompts.

51. The computing system of claim 50, wherein: the input medical query relates to a particular disease, condition, or procedure; and the first prompt instructs the sequence processing model to retrieve all mentions of the particular disease, condition, or procedure within the set of medical context data.

52. The computing system of claim 51, wherein the first prompt instructs the sequence processing model to provide direct quotations from the set of medical context data.

53. The computing system of any of claims 49-52, wherein the set of medical context data comprises data associated with at least two different data modalities.

54. The computing system of any of claims 49-53, wherein the set of medical context data comprises both image data and text data.

55. The computing system of any of claims 49-54, wherein the set of medical context data comprises a plurality of electronic health records.

56. The computing system of any of claims 49-55, wherein the input medical query comprises a diagnostic query, and wherein the output response comprises an output diagnosis.

57. One or more non-transitory computer-readable media that store instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations, the operations comprising: obtaining, by the computing system, an input medical query and a set of medical context data;performing, by the computing system, a multi-stage chain-of-prompting approach to process the input medical query and the set of medical context data with a sequence processing model to generate an output response as an output of the sequence processing model; and providing, by the computing system, the output response in response to the input medical query; wherein performing, by the computing system, the multi-stage chain-of-prompting approach comprises iteratively prompting, by the computing system, the sequence processing model with a series of prompts that are at least in part pre-defined; and wherein the series of prompts elicits, over the series of prompts, the sequence processing model to perform a chain of analysis on the input query and the set of medical context data to generate the output response.

58. The one or more non-transitory computer-readable media of claim 57, wherein: the series of prompts comprises: at least a first prompt that instruct the sequence processing model to perform information retrieval from the set of medical context data to generate a smaller set of retrieved data; and at least a second prompt that instructs the sequence processing model to perform medical analysis on the smaller set of retrieved data; and the second prompt is subsequent to the first prompt within the series of prompts.

59. The one or more non-transitory computer-readable media of claim 58, wherein: the input medical query relates to a particular disease, condition, or procedure; and the first prompt instructs the sequence processing model to retrieve all mentions of the particular disease, condition, or procedure within the set of medical context data.

60. The one or more non-transitory computer-readable media of claim 59, wherein the first prompt instructs the sequence processing model to provide direct quotations from the set of medical context data.

61. A computer-implemented method to facilitate processing of large medical context inputs with improved accuracy, the method comprising: obtaining, by a computing system comprising one or more computing devices, an input medical query and a set of multi-modal medical context data, wherein the set of multimodal medical context data comprises data associated with at least two different data modalities; processing, by the computing system, the input medical query the set of multi-modal medical context data with a sequence processing model to generate one or more responses to the medical query as an output of the sequence processing model; wherein the sequence processing model comprises a decoder transformer model comprising a plurality of transformer blocks, wherein at least one of the plurality of transformer blocks comprises a sparse feed-forward-network layer comprises a mixture of experts networks.

62. The computer-implemented method of claim 61, wherein the input medical query relates to a particular disease, condition, or procedure.

63. The computer-implemented method of any of claims 61-62, wherein the set of multi-modal medical context data comprises both image data and text data.

64. The computer-implemented method of any of claims 61-63, wherein the set of multi-modal medical context data comprises radiograph data.

65. The computer-implemented method of any of claims 61-64, wherein the set of multi-modal medical context data comprises genomic data.

66. The computer-implemented method of any of claims 61-65, wherein the set of multi-modal medical context data comprises chemical or molecular data.

67. The computer-implemented method of any of claims 61-66, wherein the set of multi-modal medical context data comprises a plurality of electronic health records.

68. The computer-implemented method of any of claims 61-67, wherein the input medical query comprises a diagnostic query, and wherein the output response comprises an output diagnosis.

69. A computing system that processes large medical context inputs with improved accuracy, the computer system comprising one or more processors and one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising: obtaining, by the computing system, an input medical query and a set of multi-modal medical context data, wherein the set of multi-modal medical context data comprises data associated with at least two different data modalities; processing, by the computing system, the input medical query the set of multi-modal medical context data with a sequence processing model to generate one or more responses to the medical query as an output of the sequence processing model; wherein the sequence processing model comprises a decoder transformer model comprising a plurality of transformer blocks, wherein at least one of the plurality of transformer blocks comprises a sparse feed-forward-network layer comprises a mixture of experts networks.

70. The computing system of claim 69, wherein the input medical query relates to a particular disease, condition, or procedure.

71. The computing system of any of claims 69-70, wherein the set of multi-modal medical context data comprises both image data and text data.

72. The computing system of any of claims 69-71, wherein the set of multi-modal medical context data comprises radiograph data.

73. The computing system of any of claims 69-72, wherein the set of multi-modal medical context data comprises genomic data.

74. The computing system of any of claims 69-73, wherein the set of multi-modal medical context data comprises chemical or molecular data.

75. The computing system of any of claims 69-74, wherein the set of multi-modal medical context data comprises a plurality of electronic health records.

76. The computing system of any of claims 69-75, wherein the input medical query comprises a diagnostic query, and wherein the output response comprises an output diagnosis.

77. One or more non-transitory computer-readable media that store instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations, the operations comprising: obtaining, by the computing system, an input medical query and a set of multi-modal medical context data, wherein the set of multi-modal medical context data comprises data associated with at least two different data modalities; processing, by the computing system, the input medical query the set of multi-modal medical context data with a sequence processing model to generate one or more responses to the medical query as an output of the sequence processing model; wherein the sequence processing model comprises a decoder transformer model comprising a plurality of transformer blocks, wherein at least one of the plurality of transformer blocks comprises a sparse feed-forward-network layer comprises a mixture of experts networks.

78. The one or more non-transitory computer-readable media of claim 77, wherein the input medical query relates to a particular disease, condition, or procedure.

79. The one or more non-transitory computer-readable media of claim 77 or 78, wherein the set of multi-modal medical context data comprises both image data and text data.

80. The one or more non-transitory computer-readable media of claim 77, 78, or 79, wherein the set of multi-modal medical context data comprises radiograph data.

Images