Natural language processing applications using large-scale language models

Abstract

To provide a method, a processor and a system that use a large-scale model to implement specific types of tasks using a large model, without a need to retrain the model.SOLUTION: In a method for generating custom endpoints, the custom endpoints can be trained for specific types of tasks, as may be indicated by the specification of one or more guidance mechanisms. A guidance mechanism can be added to or used along with a request to guide the model in performing a type of task with respect to a string of text. An endpoint receiving such a request can perform any marshalling needed to get the request in a format required by the model, a model receiving this string can process the text according to the guidance mechanisms. Such an approach can allow for a variety of tasks to be performed by a single model.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to natural language processing applications using large language models.

Background Art

[0002] Examples of applications and implementation forms of natural language processing (NLP) techniques are generally used for various tasks such as language generation or analysis, grammar and usage checking, or content summarization, to name just a few. To provide highly accurate NLP results, it may be advantageous to use large language models trained using extremely large training sets for vocabulary and grammar. A large language model (LLM) can be a very powerful universal tool for implementing a wide range of complex NLP algorithms, but such large models are computationally expensive and may require many multi-processor servers or workstations to load and perform basic calculations, thereby making them unavailable to many potential users. Entities such as cloud providers can host LLMs for use by various different users or entities, but such implementations may be sub-optimal. For example, in specific use cases or operations, the language model may require customization, which requires the model to be trained with a certain amount of additional data particularly relevant to that use case or operation. Even in the case of large cloud providers, the size and computational cost of these models reduce the feasibility of training and hosting different large models for each such use case or operation.

Summary of the Invention

Means for Solving the Problems

[0003] <� [[ID=_{22}]]Various embodiments according to the present disclosure will be described with reference to the drawings.

Brief Description of the Drawings

[0004] [Figure 1] This figure shows exemplary language inference processing systems that can be used in various embodiments. [Figure 2] This figure shows calls directed to different endpoints and the selection of language models of different sizes, based on various implementations. [Figure 3] This figure shows an example of prompt engineering and tuning, based on at least one embodiment. [Figure 4] This figure shows different inference results for different guidance mechanisms, based on at least one embodiment. [Figure 5] This figure shows an exemplary process for performing custom inference using a global model, with at least one example. [Figure 6] This figure shows an exemplary process for generating a custom endpoint for a specific type of task to be performed, using at least one embodiment. [Figure 7] This figure shows the components of an exemplary distributed system that may be used to determine and / or perform a task, according to at least one embodiment. [Figure 8A] This figure shows the inference and / or training logic according to at least one embodiment. [Figure 8B] This figure shows the inference and / or training logic according to at least one embodiment. [Figure 9] This figure shows an exemplary data center system according to at least one embodiment. [Figure 10] This figure shows a computer system according to at least one embodiment. [Figure 11] This figure shows a computer system according to at least one embodiment. [Figure 12]This figure shows at least a portion of a graphics processor according to one or more embodiments. [Figure 13] This figure shows at least a portion of a graphics processor according to one or more embodiments. [Figure 14] This is an exemplary data flow diagram for an advanced computing pipeline, with at least one implementation example. [Figure 15] This is a system diagram for an exemplary system for training, adapting, instantiating, and deploying machine learning models in an advanced computing pipeline, with at least one embodiment. [Figure 16A] This is a data flow diagram for the process of training a machine learning model, with at least one example. [Figure 16B] This figure shows a client-server architecture for extending annotation tools with a pre-trained annotation model, based on at least one embodiment. [Modes for carrying out the invention]

[0005] The following description illustrates various embodiments. For explanatory purposes, specific configurations and details are provided to give a complete understanding of the embodiments. However, it will be apparent to those skilled in the art that embodiments may be carried out without specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the embodiments described.

[0006] The various embodiments of the method can provide the use of large-scale models for various different tasks. In particular, the various embodiments provide the ability to use a single large-scale language model (LLM) for various different natural language processing (NLP) relational inference tasks without the need to retrain a single LLM or to use multiple different customized models. Inference services that enable the use of custom endpoints may be provided, and individual endpoints may be trained for specific types of tasks so that they can be indicated by the selection or provision of one or more “guidance mechanisms”. Guidance mechanisms may include tokens, tags, weights, files, modifiers, or other types of data objects that can be added to, embedded in, or associated with a request to perform a specific action or type of action with respect to a string of text. An exemplary guidance mechanism may include, among other options, a prompt token to indicate the type of action to be performed, a retrieval set tag to indicate the data set to be used for that action, and / or adapter weights that can effectively modify the action or structure of the language model. An endpoint receiving such a request may perform marshalling and / or other actions required by the language model to obtain the request in the required text format, for example, by prepending on one or more text strings (or text prefixes) to the text format request, thereby adding a guidance mechanism to the request. In some embodiments, there may be instances of the language model of different sizes, and individual endpoints may be associated with and trained on one of these instances of a particular size, which can affect aspects such as the time and cost of performing reasoning on a given request.When inference is performed using the model, the result (in the format specified by the prompt token) may be returned to an endpoint, where marshalling and / or other processing may be performed to make the result into the required structure or format, and the result may be forwarded to the intended receiver or destination. Such global services can be used by a wide variety of different users, developers, and / or other entities, each of which may generate custom endpoints for various different use cases or types of operation. This heterogeneous set of tasks may be converted into a highly homogeneous compute stream or a sequence of homogeneous batches of tasks, which may be processed using a single large-scale language model, which may be highly efficient and computationally cost-effective.

[0007] The advantages of generating such expressions can be obtained in a variety of applications and use cases. These include, but are not limited to, use in conversational systems to understand the intent of what a person is saying so that a digital conversational agent can respond appropriately. This applies broadly to situations where a computer system interacts with a human being through oral or written communication. Such techniques can also be used to analyze written documents or recordings of human speech to determine their meaning and structure, or, for example, to find grammatical errors or to generate alternative phrasing. Such techniques can also be useful in generating written or spoken content for use, for example, in video games, for marketing copy, or in creative writing or business communication applications. Another application is conversion between different languages, for example, to translate from one language to another. Yet another application is in the context of computer programming source code, in systems that automatically generate code to perform a given task, or analyze code to understand its structure and purpose, or to detect programming errors. These language-related operations can accommodate a variety of application or system types, including one or more language-related tasks, such as those related to video conferencing, image analysis, autonomous navigation, robotics, gaming and animation, and data processing, among other options.

[0008] As will be apparent to those skilled in the art in light of the teachings and suggestions contained herein, this and other variations of such functionality may also be used within the scope of various embodiments.

[0009] Figure 1 shows an exemplary system 100 that may be used to perform various language-based tasks or operations. In this example, a user or entity may use a client device 150 (or other computerized device or system) to make a call (or request or command) to system 100 to perform those tasks or operations. The call may include a certain amount of text to be processed, as well as a specification of one or more tokens (or other objects or files) that instruct how the text should be processed. In some embodiments, an endpoint selected to receive the request may be associated with these one or more tokens so that no separate instructions are required. In this example, the text should be processed using a large language model, which can be a powerful tool for performing text-based operations as described herein. Requests from client device 150 may be received in an environment such as a data center (or a cloud computing environment or a multi-tenant resource provider environment, etc.) where shared resources are used to host one or more large language models 104. In this example, these large language models 104 may be provided as part of a large language model (LLM) inference service 102 hosted by resources in a data center. A client device 150 can call appropriate endpoints 122, 126, which can then direct text to a query service 106, which can then cause a text query or operation to be performed using the appropriate LLM 104. The results of this processing, such as inferences generated by each LLM, can then be returned to the client device 150 or directed to another appropriate receiver or destination.

[0010] However, as mentioned above, while Large-Scale Language Models (LLMs) are extremely powerful universal tools for implementing a wide range of complex NLP algorithms, previous LLM-based methods did not scale well, at least in part, due to their corresponding computational costs. Since there will be various unique use cases and behaviors provided by different users, providing a single LLM in previous systems would not enable accurate results, at least in part, due to the lack of (or impracticality of) the ability to train and host multiple independent LLMs for these various behaviors and use cases.

[0011] Accordingly, various embodiments of this disclosure can present system architectures for one or more large language models to be hosted in a data center and accessed as a service by various users, in a manner that is flexibly customizable for various use cases and operations. Such architectures can also enable such deployments to be computationally efficient for service providers. System 100, such as the system shown in Figure 1, may include components such as an LLM inference service 102, one or more inference namespaces 120, such as one namespace for each developer, user, or entity, and a training namespace 140 for training endpoints for specific tasks, operations, or use cases.

[0012] The LLM inference service 102 can receive text-based requests and generate corresponding inferences. In at least some embodiments, if there are instances of LLM of different sizes (which may have different costs for use), a request may specify which model to use, such as a smaller model that is less expensive to host or run (at least in terms of one or more of the compute, memory, or storage requirements) and takes less time to perform calculations, but is best used for simple inference tasks, or a “large” model that may be more expensive to run but provides more robust inference capabilities. Although four different sizes of models are shown, there may be any number of LLM models 104 of various different sizes used. Among other differences, there may also be other differences between model instances, such as those relating to location or availability. Requests or queries received by the query service 106 of this inference service 102 may be directed to the appropriate model instance for inference, and the results may be returned to the client device 150 or target destination (after post-processing, aggregation, or other such operations).

[0013] As described above, different requests may be associated with different tasks, actions, or use cases. Instead of training and hosting different models for these various alternative forms, this exemplary system can allow individual requests to be associated with one or more tags, tokens, weights, and / or other objects or files, which can help the inference service 102 determine how best to process the request or perform inference on the request. In this example, a request (at request or inference time) may be associated with one or more guidance mechanisms, among other options, such as at least one “prompt token” (PT), “retrieval set tag” (RST), or “adapter weight” (AW). In some embodiments, there may be a fixed set of guidance mechanism types, such as three different types, but it is not required that a given request provides three different types. These guidance mechanisms can enable various customization options without changing the compute flow of inference computation through the LLM. As will be described in more detail herein, individual guidance mechanisms can cause the LLM to return different results or types of results, among other options, such as a full sentence response, a true / false response, and / or the type of speech for each word in the text string, which may be drawn from a particular set of knowledge or information. The way these mechanisms are processed allows LLM instances to process text input using the same inference process or computation, and thus a single LLM can be used for all these customized requests. Furthermore, multiple requests with the same or different associated mechanisms can be batched together for improved efficiency. Such an inference service 102 may host LLM instances of different sizes, from small LLM instances 104 to large LLM instances 104, such as those with 1 billion, 5 billion, 40 billion, and 530 billion parameters.

[0014] To enable the same processing to be formed for requests with different associated guidance mechanisms, there may be various endpoints 122, 126 to be introduced, where each endpoint may be associated with a different guidance mechanism, or a combination of guidance mechanisms. In at least some embodiments, if there may be multiple users, developers, or entities using this system, there may be various hosted inference namespaces 120 that enable user (or developer, etc.) namespaces for the endpoints to be introduced. A given inference namespace 120 may contain all endpoints for receiving requests to perform inference using the hosted LLM 104 of the inference service 102. These endpoints may be managed, for example, by the corresponding user or developer, and in some cases may be exposed to other users, and even to the public (e.g., exposed to the open internet), for use. Each endpoint 122, 126 may consist of several associated items or embodiments, which may include a name, identifier, address, and / or uniform resource locator (URL), such as " / summarize_news_article". The endpoint may also include an indicator of the size of LLM to be used for requests sent to this endpoint, as well as one or more associated inference parameters (e.g., the number of samples, or temperature). The endpoint may also include, or be associated with, one or more guidance mechanisms 124, such as prompt tokens and retrieval set tags. In at least some embodiments, the endpoint may also be associated with the type of data marshalling 128 to be performed. In some embodiments, this may be an option provided to the user, developer, or entity creating each endpoint.For structured language protocols such as JSON or Protobuf, data marshaling selections can be made to specify how to map from input fields of a particular type to the text string to be processed by the LLM, and how to map from that text string. Marshaling can be performed using code executed within the endpoint or by a process communicating with the endpoint. A request received from the client device 150 (or other such source) for the user can be received using the load balancer or ingress layer 130, which can direct the request to the appropriate endpoints 122, 126 within the namespace 120 associated with that user. For structured incoming requests such as a JSON request received via a REST endpoint or a Protobuf request received by a GRPC endpoint, one or more marshaling components 128 or processes (which may be internal to the endpoint in some embodiments) can convert this structured request data to text according to the marshaling rules 142 used during the training and creation of the namespace 120 and / or endpoint. The request can also be bound to any associated guidance mechanisms such as prompt tokens and extraction set tags and passed to the LLM inference service 102 for processing along with the associated LLM parameters. Then, the inference-based response from the LLM inference service is marshaled into a format based on each structured language protocol (e.g., JSON or Protobuf) according to one or more output data marshaling rules and returned to the client device 150 or otherwise sent to the target receiving side or destination.

[0015] For example, to provide an appropriate set of endpoints configured for a particular set of tasks for a user or developer, such a system can also provide tools such as a developer studio that can operate in the training namespace 140. Tools such as a developer studio can provide a set of workflows for managing a set of retrieval for training a guidance mechanism (such as prompt tokens 144 or adapter weights) or for referring to an LLM. As described herein, one or more marshalling rules 142 can also be provided.

[0016] Training can involve specific training and evaluation data sets for the training module 148 with respect to a given language model and can require specification of at least some training hyperparameters (such as a learning rate or a maximum number of training iterations). This training workflow can then perform several training iterations until termination criteria are met, among other options, such as when the model converges, the loss value falls below an acceptable threshold, and / or all training data has been used. A user can perform tasks such as training and evaluation for each guidance mechanism to be used for a given task.

[0017] Once customization and training are complete, the associated customization data can be issued to a new endpoint in the corresponding inference namespace 120. The endpoint can be equipped with access control or other mechanisms or configurations such that it can be provided as part of an endpoint management system. Such developer tools can operate in various locations, among other options, such as in the cloud as part of an inference system and / or on-premises at a user location.

[0018] If this tool is provided as part of a hosted cloud service, the system may automatically recalculate prompt tokens or adapter weights from the associated training data and automatically reintroduce the associated endpoints when the underlying LLM is updated. At any time, the user can download the associated guidance mechanisms and use them for inference with the associated LLMs within the user's location without going through the inference namespace 120. The Developer Studio may also be configured to implement additional functionality to conserve processing time. For example, an endpoint may be trained for a small LLM for a given prompt token, result set tags, and / or adapter weights. Since the training process does not require much additional processing when done together, the Studio may also generate endpoints for other sizes of LLMs, such as intermediate or large LLMs, or at least the next largest LLMs, for the same guidance mechanism. Creating new endpoints from scratch can be quite resource-intensive, and therefore there may be advantages in training and creating them together for model instances of different sizes. The Developer Studio may also be configured to automatically generate updated endpoints for updates or modifications of each language model.

[0019] Figure 2 shows an exemplary call flow 200 in such a system. In this example, there are two different client devices 202, 204 which may be associated with the same or different entities, and client devices 202, 204 may make calls to the same or different namespaces, as described with respect to Figure 1. In some embodiments, there may also be a general-purpose or "public" namespace with a common prompt token and / or a publicly available set of retrieves which may be used by multiple different entities. In this example, client A 202 has a string of text which should be processed using the LLM of the inference service 216. Based at least on the intended use of the result of processing, the request may have one or more guidance mechanisms 218 associated with the request, such as a specific prompt token to identify the type of result which should be returned, as well as a set of retrieves to identify a specific database which the LLM should use to process the request. In this example, there would be a trained endpoint, here in particular endpoint A 204 which is trained on the text associated with these particular guidance mechanisms 218. That endpoint can take any suitable form, such as the form of a REST endpoint. Generally, there will be other endpoints, such as endpoint B 208, which may be in the same namespace, and these endpoints will not be called on this request because they are trained for different guidance mechanisms. Endpoints can also accommodate different types, and therefore, a subset of endpoints may be REST endpoints, and another subset may be endpoints of other types, whether they are in the same namespace or a different namespace. The called endpoint, in this case endpoint A 206, can use these guidance tokens, along with its associated guidance mechanism, to transform the input request, which contained the text to be processed, into a request that is in a format suitable for processing by the generic LLM of the inference service 216.In some embodiments, a client may not, for example, call a specific endpoint but may submit a request to a generic interface or API, which can analyze the request and direct it to the appropriate endpoint. Each endpoint may also be associated with or trained on an LLM of a specific size; therefore, endpoint A 206 may be trained, configured, or customized to send this formatted string, to be processed by a larger LLM 212, to the LLM inference service 216. The inference results in this example may be returned to client A 202 via endpoint A 206, but in other examples, the results may be passed through different endpoints or sent to different (or additional) receivers or destinations. Although not illustrated, the processing results may also be stored by or for the LLM inference service for use in further training of the LLM model.

[0020] As shown, a separate client device B 204 may submit a request containing a different set of text to be processed and guidance mechanisms 220, which will then be directed to different endpoints C 210 associated with those guidance mechanisms. If clients B 204 and A 202 are associated with a single entity, these endpoints 206, 208, and 210 may all be associated with a single namespace; otherwise, at least endpoint A 206 and endpoint C 210 may be in different namespaces. In this example, endpoint C 210 then directs the formatted request to a target LLM, which may be one of the LLMs, such as the smaller LLM 214 described, but also to a larger LLM 212, since in this example a single LLM model may be shared by multiple different entities for multiple different types of natural language processing (or other such) tasks. The result may be passed through endpoint C 210 (or another endpoint) to client device B 204 and / or another target destination or receiver. Although not shown in Figure 2, the endpoint (or the process communicating with the endpoint) can also perform the marshalling described with respect to Figure 1, the request received from the client may be in a structured format such as JSON, and that request may be converted to text to generate a text request to be provided to the LLM, the response received from the LLM (and inference service) may be in text format, and that response may also undergo the marshalling process, so that the response received by the requesting client device is in an appropriate structured format.

[0021] In at least one embodiment, a prompt token may be used to provide guidance or instructions on how the associated text should be processed. The prompt token may take the form of a series of numbers that may be prepended to (or otherwise inserted into or after) the input request text string. The series of numbers (or alphanumeric characters) may not be human-readable, but may provide guidance on how the subsequent text in the string should be processed. The text string may be a sentence in natural language (e.g., American English). Regardless of the type of processing to be performed, the input to the LLM may be a natural language text string prepended by an alphanumeric sequence type prefix indicating the type of inference to be performed or the type of response to be returned.

[0022] Figure 3 shows an exemplary stage 300 in training and / or generating an exemplary prompt token. In this example, a training text input string 302 is provided, which presents the question, “Which country is Normandy located in?” During prompt engineering stage 304, a prompt token is generated that corresponds to the command “Answer the question.” It should be understood that while the prompt is presented here in human-readable text, in many cases, prompt tokens may take the form of alphanumeric strings that are not human-readable. During the training process, the prompt token generator can learn to generate appropriate prompt tokens to accomplish the target task by analyzing training data associated with this type of task, such as input text and corresponding prompt tokens. As mentioned above, there may be one prompt token stored or associated with each endpoint, and each endpoint may, in at least some embodiments, be its own API. In addition to generating a representative string that should act as a prompt token, prompt engineering stage 304 can also determine and store a routine for extracting results from the generated text.

[0023] The prompt in this example may also go through a prompt tuning stage 310. The prompt tuning stage may involve a process that attempts to improve the results of inference by the LLM, which are “frozen” or fixed and generally cannot be customized for a given task, by improving the quality of the process associated with the prompt token so that more accurate results can be obtained for a particular task. In this example, the input task may be tokenized 314 in order to analyze its components in a finer-grained manner, which may help generate one or more “soft” prompt tokens 312. Each soft prompt token 316 may modify something about the primary prompt token, or a set of soft prompt tokens 312 may be used in place of a single prompt token 306. The soft prompt tokens 312 may take the form of virtual tokens, each of which can instruct a more specific command. Instead of simply instructing the LLM to answer a question, one may instruct the manner of answering, the emotional state to be conveyed, the format of the answer, etc. These prompts can take various forms in some embodiments, including text prompts, learned embeddings from prompt tuning, and / or learned prompt models. The returned results may be in a specified format, as indicated by guidance tokens, such as a single boolean value, structured formatted text, or multiple responses about the sound type of each word in the input text string. In some embodiments, the results may also provide decisions made regarding the input text, such as the sentiment expressed by the movie review.

[0024] Since LLMs speak only text, the instructions provided must also be in text format. Prompt tokens can take the form of prefixes in text that provide some instruction or “guidance” regarding the reasoning to be performed. As mentioned above, this could be, among other options, instructing the model to answer questions in true / false format, infer the sentiment of text, determine whether a sentence is logical, extract a specific type of information (e.g., name, address, or title, if present), summarize a paragraph, suggest a subject for the body of text, or answer questions using a library specified by an extract set tag. An LLM can be thought of as a model that understands natural language, and it may be asked to do several different things, as long as the requests are also in that natural language and the model is trained to understand the requests. During training, the model may be able to evaluate examples of the type of task to be performed and generate sequences of numbers (or alphanumeric characters) that can be obtained based on a language or syntax that a computer or network can understand, which can be understood by the model. The results from the model may be evaluated, and if those results are inaccurate or not of the desired type, further training may be performed to attempt to improve the instruction string used for the prompt token. Processes such as P-tuning may be used to improve this token.

[0025] Adapter weights can be inserted into the text of a request to be sent to an inference service or language model, for example, by prepending them on the text string to be processed, as with other guidance mechanisms. In some embodiments, the guidance mechanisms should be attached in a specific order, and in other embodiments, the ordering may not be fixed, but each guidance mechanism may include an indicator (e.g., a specific code or symbol) that identifies the portion of the text string corresponding to a particular guidance mechanism. Adapter weights can act as perturbations to the activation of the LLM, which can cause the LLM to return different types of results for different requests with different adapter weights. Adapter weights can be thought of as adjustments to one or more of the network weights, which can effectively cause the network to perform inference in slightly different ways without requiring retraining to adjust the associated network weights (one or more). A training process, such as the training process described with respect to Figure 1, can be used to determine appropriate adapter weights for a given task, i.e., those that should be associated with each endpoint. Exemplary adapter weights can indicate that further layer processing should be performed between two layers of the model, such as between layer 37 and layer 38, with specific weights or activations. Such a technique can effectively modify the model's behavior without the need for retraining.

[0026] Retrieval set tags may be used to specify a database, dictionary, library, or other set of data or facts to be used for a particular inference task associated with an endpoint. The advantage of using such tags is that they can allow a language model to use data other than that used to train it, effectively expanding the model's capabilities without requiring retraining with this data. Two different requests may point to two different databases with different sets of facts, and thus the model may answer the exact same question with two different answers. Anything else related to these requests, other than database lookups, may be computationally equivalent or homogeneous, which helps to make this process highly efficient. The format of the retrieval set can vary, as long as the data is in a text format that is understandable to and accessible by the language model or inference service in question. In some embodiments, some amount of preprocessing may be performed, in which data is extracted from the database or fact set indicated by the retrieval set tag, and the data is formatted and indexed in a specified format to be provided as input to the language model. Such techniques can allow a given database to be more varied in structure or form, insofar as the data itself can be made into a single format for use by the language model. In some embodiments, preprocessing can effectively index the fact set, among other options, and provide that index to the model, so that the language model or the inner layers of the network can access the facts (or documents, etc.) identified through the index.A database of legal terminology may be useful for a different task than a set of recipes, and generally, combining these facts into a single database would not be beneficial, as that database would either become excessively large or require a reduction in the number of facts of a given type to be practical, which would degrade the quality of the results. Therefore, different data sets may be better suited to specific tasks. In some cases, there may be four different endpoints, which are identical except that they use different sets of facts and are trained or customized for those different sets of facts.

[0027] Figure 4 shows a set of different types of inference results 400 that may be returned for the same text string using the same language model, but with different custom endpoints. While “endpoint” is used as the primary example herein, it should be understood that there are any number of ways to cause a particular or custom processing to be applied to an incoming request using various models, rules, identifiers, interfaces, and / or other such options, and therefore “endpoint” (whether REST or otherwise) should not be considered a limitation to the scope of various embodiments. In this example, a single text string 402 may be received by multiple different endpoints, which may be associated with different guidance mechanisms or combinations of guidance mechanisms. Different types of results 404 may be returned by the use of different prompt tokens or other such guidance mechanisms. In this example, for the text string “Is the Earth Round?”, the results may have similar content, but in different forms, such as “Yes” for a Yes / No result, “True” for a True / False result, “The Earth is Round” for a “Answer the question in Natural Language” result, or “1” for a binary 1 / 0 result. Another result for different prompt tokens returns the part of speech of the text string, identified at least using a specific set of definitions. Another result demonstrates the difference in results based on specifying the fact set, such as if a modern fact set were asked whether the Earth is round, it might return the answer "yes," but if someone is deciding the response based on a fact set such as religious texts from medieval Europe, result 406 could be "no" in the case of a yes / no result, since that is the correct answer by that particular fact set. As shown, a single large language model can be guided to return entirely different results by using different guidance mechanisms.

[0028] Such methods can enable multiple users to effectively customize the performance of a model, such as an LLM, without actually modifying or retraining the model. These users can instead use customized endpoints that can format text processing requests through the use of one or more guidance mechanisms, so that the LLM knows how to handle this particular request to perform a specific action or type of task. Such a model can then be used simultaneously by multiple different users to perform multiple different types of tasks. As mentioned above, for efficiency purposes, these similarly formatted requests can also be batched together for transmission and processing, but they may correspond to different types of tasks to be performed or results to be provided. The actions performed by the LLM may not change as a result of these guidance mechanisms, and therefore a single model performing the same (or similar) calculations may be used without retraining or customization unless desired for improvement over time or for another such purpose. For example, if a retrieve set tag instructs the use of a specific database, the model will perform slightly different calculations. However, it may be possible to reduce tens of thousands of customized requests to a single computation to be performed, or a small set of computations (e.g., 2 to 10). Essentially, by at least one embodiment, such a system or service may take a heterogeneous set of tasks, which may be converted into a homogeneous computation string that can be handled by a single model or instances of the same model, making the system or service far more efficient and computationally cost-effective.

[0029] Figure 5 shows an exemplary process 500 for causing a large-scale language model to perform a custom inference task that may be performed, in various embodiments. It should be understood that, with respect to this and other processes described herein, unless otherwise specified, there may be additional, fewer, or alternative operations performed in a similar or alternative order, or at least partially in parallel, within the scope of various embodiments. In this example, in 502, a request is received to an endpoint associated with a specific natural language processing (NLP) relational operation, such as a specific inference task to be performed to return a particular type of result. The request in this example may contain structured text, such as text in JSON or a similar format. In 504, the structured text, if present, may be converted to unstructured text in a natural language format (or other text format used by the target language model), for example, by using a marshalling / unmarshalling process specific to that endpoint. The endpoint may also, in 506, append one or more alphanumeric prefixes corresponding to one or more guidance mechanisms as prefixes (or other additions) to the text string to be processed. As described herein, this may include adding one or more alphanumeric strings that in some way instruct or modify the behavior of a large language in order to obtain a particular type of inference result. This text request may be forwarded for processing or inference to a large language model (or, for example, an inference service hosting such a model), where the model to which the request is forwarded may be an instance of a particular size associated with the endpoint. Then, in 510, the large language model may be prompted to perform inference on the text string in the request as directed by one or more guidance mechanisms. Then, in 512, the inference result may be received by the endpoint, which may, in 514, convert the text into a structured format if necessary.Next, in 516, the result may be forwarded to a target receiver or destination for the result, which may be different from or the same as the source of the request. As described above, in some embodiments, this request may be batched with other requests or added to a single result stream, which can help improve operational efficiency.

[0030] Figure 6 shows an exemplary process 600 for generating task-specific endpoints for a trained model that can be implemented in various embodiments. In this example, at 602, a request is received (to the Developer Studio or other such system or service) to generate a custom endpoint for a type of natural language task to be implemented using a large-scale language model. At the endpoint, several operations may be performed, which are shown sequentially, but may also be performed simultaneously or in other orders as appropriate. In one operation, at 604, it may be determined whether there are marshalling rules (or other format or processing rules) to be applied to format the incoming request in a target format for the language model, such as an unstructured natural language format. Other formats or rules may be used for other types of models or tasks. For a given type of task, at 606, a prompt token may be determined that can indicate the type of inference to be performed and / or the type or result to be returned. In some cases, this may involve a generation and / or training process that can provide tokens that can guide the language model when performing the target task. A prompt token can take one of several different forms, but in a language processing model, it can take the form of an alphanumeric string that provides a code that the language model can understand in order to perform a task of a target type. In 608, if a particular set of facts should be used for a task, a pull-set tag may be determined that can instruct the model which set of facts should be used for inference. As mentioned above, such a tag does not affect the process to be performed, but instead indicates the set of data or content from which the model should pull when performing inference, such as to determine the answer to a question.During this training process, it may also be learned that some minor modification to the behavior of the language model is needed or at least advantageous, where such modification may be carried out through the introduction of one or more adapter weights that can effectively modify one or more network weights or network layers of the model. In 610, if so, these weights may be determined or learned throughout the training process and provided for use when generating endpoints. In some cases, there will be a particular model, type of model, and / or size of model to which the endpoint is trained, where so, in 612, this model may be identified, and therefore the endpoint is trained to work with respect to that particular model instance. Using this information, in 614, a custom endpoint may be generated during this training or generation process, which can translate incoming requests according to provided marshalling (or other) rules, causing a directed instance of the language model to perform inference according to the identified guidance mechanism, for example, by prepending a text string about the guidance mechanism to the request produced by the custom endpoint. Subsequently, in 616, the custom endpoint may be provided for use in causing the language model (or any other such model or network) to perform a particular type of inference or return a particular type of result, without requiring the retraining of the large language model or the generation of a new model. In at least some embodiments, the endpoint may be introduced into a namespace associated with the entity that generated the endpoint.

[0031] As described herein, the various methods presented herein are lightweight enough to be performed in real time or near real time on a client device such as a personal computer or gaming console. Such processing may be performed on content generated on the client device or received from an external source, such as streaming content received over at least one network. The source may be any suitable source, among other options, such as a game host, a streaming media provider, a third-party content provider, or another client device. In some cases, the processing and / or rendering of this content may be performed by one of these other devices, systems, or entities and then provided to the client device (or another such recipient) for presentation or another such use.

[0032] As an example, Figure 7 shows an exemplary network configuration 700 that may be used to provide, generate, modify, encode, and / or transmit content. In at least one embodiment, a client device 702 may generate or receive content for a session using components of a content application 704 on the client device 702 and data stored locally on that client device. In at least one embodiment, a content application 724 (e.g., a text processing application) running on a content server 720 (e.g., a cloud server or edge server) may initiate a session associated with at least a client device 702, using a session manager and user data stored in a user database 734, causing content 732 to be determined by a content manager 726. The content generation or management application 726 may generate or receive requests to perform NLP relational processing, which may be passed to a large language model 730 for processing, such as being provided through an inference service hosted using resources on the server. In some embodiments, the request may be directed to a custom endpoint 728 associated with a particular type of task, which may then be performed by a model. At least a portion of the generated content (e.g., inference results, or data for the custom endpoint) may be sent to a client device 702 using a suitable transmission manager 722 for download, streaming, or other such transmission channel. An encoder may be used to encode and / or compress at least a portion of this data before sending it to the client device 702.In at least one embodiment, a client device 702 receiving such content may provide this content to a corresponding content application 704, which may likewise include, or alternatively, a content manager 710, a custom endpoint 712, or a developer studio 714 for use in generating custom endpoints, among other options. A decoder may also be used to decode data received over one or more networks 740 for presentation via the client device 702, such as image or video content through a display 706, and audio such as sound and music through at least one audio playback device 708, such as speakers or headphones. In at least one embodiment, at least a portion of this content may already be stored on the client device 702, rendered on the client device 702, or accessible to the client device 702, such that transmission over the network 740 is not required for at least that portion of the content, for example, if the content has been previously downloaded or stored locally on a hard drive or optical disc. In at least one embodiment, a transmission mechanism such as data streaming may be used to transfer this content from server 720 or user database 734 to client device 702. In at least one embodiment, at least a portion of this content may be obtained from or streamed from another source, such as a third-party service 760 which may also include a content application 762 for generating or providing content. In at least one embodiment, this functional portion may be implemented using multiple computing devices or multiple processors in one or more computing devices, such as including a combination of CPU and GPU.

[0033] In this example, these client devices may include any suitable computing device, such as a desktop computer, notebook computer, set-top box, streaming device, gaming console, smartphone, tablet computer, VR headset, AR goggles, MR headset / goggle wearable computer, or smart television. Each client device may submit requests over at least one wired or wireless network, such as the Internet, Ethernet, local area network (LAN), or cellular network, among other options. In this example, these requests may be submitted to an address associated with the cloud provider that can operate or control one or more electronic resources in the cloud provider environment, such as a data center or server farm. In at least one embodiment, the requests may be received or processed by at least one edge server located on the network edge and outside at least one security layer associated with the cloud provider environment. In this way, latency can be reduced by enabling client devices to interact with servers that are closer, while also improving the security of resources in the cloud provider environment.

[0034] In at least one embodiment, such a system may be used to perform graphical rendering operations. In other embodiments, such a system may be used for other purposes, such as providing image or video content to test or validate autonomous machine applications, or to perform deep learning operations. In at least one embodiment, such a system may be implemented using edge devices or incorporating one or more virtual machines (VMs). In at least one embodiment, such a system may be implemented at least partially in a data center or at least partially using cloud computing resources.

[0035] Reasoning and training logic Figure 8A shows the inference and / or training logic 815 used to perform the inference and / or training operations associated with one or more embodiments. Further details regarding the inference and / or training logic 815 are provided below in conjunction with Figures 8A and / or 8B.

[0036] In at least one embodiment, the inference and / or training logic 815 may include, but not limited to, code and / or data storage 801 for storing forward and / or output weights and / or input / output data, and / or other parameters, for constituting neurons or layers of a neural network used for training and / or inference in one or more embodiments. In at least one embodiment, the training logic 815 may include, or be coupled to, code and / or data storage 801 for storing graph code or other software for controlling timing and / or sequence, and weight and / or other parameter information should be loaded into the code and / or data storage 801 to constitute logic including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, the code, such as graph code, loads weight or other parameter information into the processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, the code and / or data storage 801 stores the weight parameters and / or input / output data of each layer of the neural network being trained or used in conjunction with one or more embodiments during the forward propagation of input / output data and / or weight parameters during training and / or inference using the embodiments of one or more embodiments. In at least one embodiment, any portion of the code and / or data storage 801 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0037] In at least one embodiment, any portion of the code and / or data storage 801 may be inside or outside one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or code and / or data storage 801 may be cache memory, dynamic randomly addressable memory ("DRAM"), static randomly addressable memory ("SRAM"), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the selection of whether the code and / or code and / or data storage 801 is inside or outside the processor, or whether it consists of DRAM, SRAM, flash, or some other type of storage, may depend on available storage, on-chip vs. off-chip, latency requirements of the training and / or inference functions being performed, batch size of data used in neural network inference and / or training, or any combination of these factors.

[0038] In at least one embodiment, the inference and / or training logic 815 may include code and / or data storage 805 for storing backward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network used to train and / or infer in one or more embodiments. In at least one embodiment, the code and / or data storage 805 stores weight parameters and / or input / output data for each layer of the neural network used to train or in conjunction with one or more embodiments during backward propagation of input / output data and / or weight parameters during training and / or inference using one or more embodiments. In at least one embodiment, the training logic 815 may include, or be coupled to, code and / or data storage 805 for storing graph code or other software for controlling timing and / or sequence, and weight and / or other parameter information should be loaded into the code and / or data storage 801 to constitute logic including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, the code, such as graph code, loads weight or other parameter information into the processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, any portion of the code and / or data storage 805 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of the code and / or data storage 805 may be inside or outside one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 805 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage.In at least one embodiment, the selection of whether the code and / or data storage 805 is, for example, internal or external to the processor, or whether it consists of DRAM, SRAM, flash or some other type of storage, may depend on the available storage, on-chip vs. off-chip, latency requirements of the training and / or inference functions being performed, the batch size of the data used in the neural network inference and / or training, or any combination of these factors.

[0039] In at least one embodiment, the code and / or data storage 801 and the code and / or data storage 805 may be separate storage structures. In at least one embodiment, the code and / or data storage 801 and the code and / or data storage 805 may be the same storage structure. In at least one embodiment, the code and / or data storage 801 and the code and / or data storage 805 may be partially the same storage structure and partially separate storage structures. In at least one embodiment, any portion of the code and / or data storage 801 and the code and / or data storage 805 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0040] In at least one embodiment, the inference and / or training logic 815 may include, but not limited to, one or more arithmetic logic units ("ALUs") 810, including integer and / or floating-point units, for performing logical and / or mathematical operations that are at least partially based on or directed by training and / or inference code (e.g., graph code), the results of which may produce activations (e.g., output values ​​from layers or neurons in a neural network) stored in activation storage 820, the activations being functions of input / output and / or weight parameter data stored in code and / or data storage 801 and / or code and / or data storage 805. In at least one embodiment, the activation stored in the activation storage 820 is generated according to linear algebra and / or matrix-based mathematics performed by (one or more) ALU 810 in response to the execution of an instruction or other code, and the weight values ​​stored in the code and / or data storage 805 and / or code and / or data storage 801 are used as operands along with other values ​​such as bias values, gradient information, momentum values, or other parameters or hyperparameters, any or all of which may be stored in the code and / or data storage 805 or code and / or data storage 801, or in other on-chip or off-chip storage.

[0041] In at least one embodiment, the (one or more) ALU 810 are contained within one or more processors or other hardware logic devices or circuits, but in another embodiment, the (one or more) ALU 810 may be outside of the processors or other hardware logic devices or circuits (e.g., coprocessors) that use them. In at least one embodiment, the ALU 810 may be contained within an execution unit of a processor, or otherwise contained within a bank of ALUs accessible by execution units of a processor, either within the same processor or distributed across different types of processors (e.g., a central processing unit, a graphics processing unit, a fixed-function unit, etc.). In at least one embodiment, the code and / or data storage 801, the code and / or data storage 805, and the activation storage 820 may be on the same processor or other hardware logic device or circuit, but in another embodiment, they may be in different processors or other hardware logic devices or circuits, or in some combination of the same processor or other hardware logic devices or circuits and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of the activated storage 820 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. Furthermore, inference and / or training code may be stored together with other code accessible to the processor or other hardware logic or circuitry, and may be fetched and / or processed using the processor's fetch, decode, schedule, execute, retirement, and / or other logic circuits.

[0042] In at least one embodiment, the activated storage 820 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the activated storage 820 may be entirely or partially located within or outside one or more processors or other logic circuits. In at least one embodiment, the selection of whether the activated storage 820 is, for example, inside or outside the processor, or whether it consists of DRAM, SRAM, flash, or some other type of storage, may depend on the available storage, on-chip vs. off-chip, latency requirements of the training and / or inference functions being performed, batch size of data used in neural network inference and / or training, or any combination of these factors. In at least one embodiment, the inference and / or training logic 815 shown in Figure 8a may be used in conjunction with an application-specific integrated circuit ("ASIC"), such as a Tensorflow® processing unit from Google, an inference processing unit (IPU) from Graphcore®, or a Nervana® (e.g., "Lake Crest") processor from Intel Corp. In at least one embodiment, the inference and / or training logic 815 shown in Figure 8a may be used in conjunction with other hardware, such as a central processing unit ("CPU"), a graphics processing unit ("GPU"), or a field-programmable gate array ("FPGA").

[0043] Figure 8b shows the inference and / or training logic 815 in at least one or more embodiments. In at least one embodiment, the inference and / or training logic 815 may include, but is not limited to, hardware logic in which computational resources are either dedicated or used only in conjunction with weight values ​​or other information corresponding to one or more layers of neurons in a neural network. In at least one embodiment, the inference and / or training logic 815 shown in Figure 8b may be used in conjunction with application-specific integrated circuits (ASICs), such as a Tensorflow® processing unit from Google, an Inference Processing Unit (IPU) from Graphcore®, or a Nervana® (e.g., "Lake Crest") processor from Intel Corp. In at least one embodiment, the inference and / or training logic 815 shown in Figure 8b may be used in conjunction with other hardware, such as a central processing unit (CPU) hardware, a graphics processing unit (GPU) hardware, or a field-programmable gate array (FPGA). In at least one embodiment, the inference and / or training logic 815 includes, but is not limited to, code and / or data storage 801 and code and / or data storage 805, which may be used to store code (e.g., graph code), weight values, and / or other information including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. In at least one embodiment shown in Figure 8b, each of the code and / or data storage 801 and code and / or data storage 805 is associated with a dedicated computing resource, such as compute hardware 802 and compute hardware 806, respectively. In at least one embodiment, each of the compute hardware 802 and compute hardware 806 comprises one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on the information stored in the code and / or data storage 801 and code and / or data storage 805, respectively, and the results are stored in activation storage 820.

[0044] In at least one embodiment, each of the code and / or data storages 801 and 805 and the corresponding compute hardware 802 and 806 correspond to different layers of a neural network, thereby the activation resulting from one “storage / compute pair 801 / 802” of the code and / or data storage 801 and compute hardware 802 is provided as input to the “storage / compute pair 805 / 806” of the code and / or data storage 805 and compute hardware 806 in order to mirror the conceptual organization of the neural network. In at least one embodiment, each of the storage / compute pairs 801 / 802 and 805 / 806 may correspond to two or more neural network layers. In at least one embodiment, additional storage / compute pairs (not shown) may be included in the inference and / or training logic 815 after or in parallel with the storage / compute pairs 801 / 802 and 805 / 806.

[0045] Data center Figure 9 shows an exemplary data center 900 in which at least one embodiment may be used. In at least one embodiment, the data center 900 includes a data center infrastructure layer 910, a framework layer 920, a software layer 930, and an application layer 940.

[0046] In at least one embodiment, as shown in Figure 9, the data center infrastructure layer 910 may include a resource orchestrator 912, grouped computing resources 914, and node computing resources ("node CRs") 916(1) to 916(N), where "N" represents any positive integer. In at least one embodiment, the node CRs 916(1) to 916(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processors, etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules, and cooling modules, etc. In at least one embodiment, one or more nodes CR from among nodes CR916(1) to 916(N) may be servers having one or more of the computing resources described above.

[0047] In at least one embodiment, the grouped computing resources 914 may include separate groupings of node CRs housed in one or more racks (not shown), or many racks housed in a data center at various geographical locations (also not shown). A separate grouping of node CRs within the grouped computing resources 914 may include grouped compute resources, network resources, memory resources, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several node CRs, including CPUs or processors, may be grouped in one or more racks to provide compute resources to support one or more workloads. In at least one embodiment, one or more racks may also include any number of power modules, cooling modules, and network switches in any combination.

[0048] In at least one embodiment, the resource orchestrator 912 may constitute or otherwise control one or more nodes CR916(1) to 916(N) and / or a grouped computing resource 914. In at least one embodiment, the resource orchestrator 912 may include a software design infrastructure ("SDI") management entity for the data center 900. In at least one embodiment, the resource orchestrator may include hardware, software, or any combination thereof.

[0049] In at least one embodiment, as shown in Figure 9, the framework layer 920 includes a job scheduler 922, a configuration manager 924, a resource manager 926, and a distributed file system 928. In at least one embodiment, the framework layer 920 may include a framework for supporting software 932 of the software layer 930 and / or one or more applications 942 of the application layer 940. In at least one embodiment, the software 932 or (one or more) applications 942 may include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure, respectively. In at least one embodiment, the framework layer 920 may be a type of free and open-source software web application framework, such as Apache Spark® ("Spark"), which may use the distributed file system 928 for large-scale data processing (e.g., "big data"). In at least one embodiment, the job scheduler 922 may include a Spark driver to facilitate scheduling of workloads supported by various layers of the data center 900. In at least one embodiment, the configuration manager 924 may be able to configure different layers, such as the software layer 930 and the framework layer 920, which includes Spark and a distributed file system 928 to support large-scale data processing. In at least one embodiment, the resource manager 926 may be able to manage clustered or grouped computing resources that are mapped or allocated to support the distributed file system 928 and the job scheduler 922. In at least one embodiment, the clustered or grouped computing resources may include grouped computing resources 914 in the data center infrastructure layer 910.In at least one embodiment, the resource manager 926 may work in conjunction with the resource orchestrator 912 to manage these mapped or allocated computing resources.

[0050] In at least one embodiment, the software 932 contained within the software layer 930 may include software used by nodes CR916(1) to 916(N), grouped computing resources 914, and / or at least a portion of the distributed file system 928 of the framework layer 920. One or more types of software may include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0051] In at least one embodiment, one or more applications 942 contained within the application layer 940 may include one or more types of applications used by nodes CR916(1) to 916(N), grouped computing resources 914, and / or at least a portion of the distributed file system 928 of the framework layer 920. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive compute, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0052] In at least one embodiment, any of the configuration manager 924, resource manager 926, and resource orchestrator 912 may implement any number and type of self-correcting actions based on any amount and type of data obtained in any technically feasible manner. In at least one embodiment, the self-correcting actions may relieve the data center operator of data center 900 of the task of determining potentially faulty configurations and potentially avoiding underutilized and / or underperforming portions of the data center.

[0053] In at least one embodiment, the data center 900 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by computing weight parameters according to a neural network architecture using the software and computing resources described above with respect to the data center 900. In at least one embodiment, a trained machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to the data center 900 by using weight parameters computed through one or more training techniques described herein.

[0054] In at least one embodiment, the data center may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, or other hardware to perform training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above may be configured as services that enable users to train information or perform inference on information, such as image recognition, speech recognition, or other artificial intelligence services.

[0055] The inference and / or training logic 815 is used to perform inference and / or training operations associated with one or more embodiments. Details relating to the inference and / or training logic 815 are provided below in conjunction with Figures 8a and / or 8b. In at least one embodiment, the inference and / or training logic 815 may be used in the system of Figure 9 for inference or prediction operations, at least in part, based on weight parameters calculated using the neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0056] Such components can be used to enable custom tasks to be performed using a model, such as a large-scale language model, without the need to retrain the same model.

[0057] Computer system Figure 10 is a block diagram showing an exemplary computer system, which may be a system 1000 having interconnected devices and components, a system-on-a-chip (SOC), or any combination thereof, formed together with a processor that may include an execution unit for executing instructions, according to at least one embodiment. In at least one embodiment, the computer system 1000 may include components such as a processor 1002 for employing an execution unit including logic for implementing algorithms for process data, as described herein, but not limited to the embodiments described herein. In at least one embodiment, computer system 1000 may include processors such as the PENTIUM® processor family, Xeon®, Itanium®, XScale®, and / or StrongARM®, Intel® Core®, or Intel® Nervana® microprocessors available from Intel Corporation in Santa Clara, California, but other systems (including PCs with other microprocessors, engineering workstations, set-top boxes, etc.) may also be used. In at least one embodiment, computer system 1000 may run a version of the WINDOWS® operating system available from Microsoft Corporation in Redmond, Washington, but other operating systems (e.g., UNIX® and Linux®), embedded software, and / or graphical user interfaces may also be used.

[0058] The embodiments may be used in other devices, such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol devices, digital cameras, personal digital assistants ("PDAs"), and handheld PCs. In at least one embodiment, the embedded application may include a microcontroller, a digital signal processor ("DSP"), a system-on-a-chip, a network computer ("NetPC"), a set-top box, a network hub, a wide area network ("WAN") switch, or any other system capable of implementing one or more instructions according to at least one embodiment.

[0059] In at least one embodiment, the computer system 1000 may include, but is not limited to, a processor 1002, which may include, but is not limited to, one or more execution units 1008 for performing machine learning model training and / or inference by the techniques described herein. In at least one embodiment, the computer system 1000 is a single-processor desktop or server system, while in another embodiment, the computer system 1000 may be a multi-processor system. In at least one embodiment, the processor 1002 may include, but is not limited to, a complex instruction set computer ("CISC") microprocessor, a reduced instruction set computing ("RISC") microprocessor, a very long instruction word ("VLIW") microprocessor, a processor implementing a combination of instruction sets, or any other processor device, such as a digital signal processor. In at least one embodiment, the processor 1002 may be coupled to a processor bus 1010, and the processor bus 1010 may transmit data signals between the processor 1002 and other components in the computer system 1000.

[0060] In at least one embodiment, the processor 1002 may include, but is not limited to, a level 1 ("L1") internal cache memory ("cache") 1004. In at least one embodiment, the processor 1002 may have a single internal cache or multiple levels of internal caches. In at least one embodiment, the cache memory may reside outside the processor 1002. Other embodiments may also include a combination of both internal and external caches, depending on the specific implementation and requirements. In at least one embodiment, the register file 1006 may store different types of data in various registers, including, but is not limited to, integer registers, floating-point registers, status registers, and instruction pointer registers.

[0061] In at least one embodiment, but not limited to, an execution unit 1008 containing logic for performing integer and floating-point arithmetic is also present in the processor 1002. In at least one embodiment, the processor 1002 may also include a microcode ("u-code") read-only memory ("ROM") for storing microcode for several macro instructions. In at least one embodiment, the execution unit 1008 may include logic for handling a packed instruction set 1009. In at least one embodiment, by including the packed instruction set 1009, along with the associated circuit elements for executing the instructions, in the instruction set of the general-purpose processor 1002, arithmetic used by many multimedia applications can be performed using packed data in the general-purpose processor 1002. In one or more embodiments, many multimedia applications may be accelerated and run more efficiently by using the full width of the processor's data bus to perform arithmetic on packed data, which may eliminate the need to transfer smaller units of data across the processor's data bus to perform one or more arithmetic operations, one data element at a time.

[0062] In at least one embodiment, the execution unit 1008 may also be used in a microcontroller, embedded processor, graphics device, DSP, and other types of logic circuits. In at least one embodiment, the computer system 1000 may include, but is not limited to, memory 1020. In at least one embodiment, memory 1020 may be implemented as a dynamic random access memory ("DRAM") device, a static random access memory ("SRAM") device, a flash memory device, or other memory device. In at least one embodiment, memory 1020 may store (one or more) instructions 1019 and / or data 1021, which are represented by data signals that can be executed by the processor 1002.

[0063] In at least one embodiment, a system logic chip may be coupled to the processor bus 1010 and memory 1020. In at least one embodiment, the system logic chip may include, but is not limited to, a memory controller hub ("MCH") 1016, and the processor 1002 may communicate with the MCH 1016 via the processor bus 1010. In at least one embodiment, the MCH 1016 may provide a high-bandwidth memory path 1018 to memory 1020 for instruction and data storage, as well as for the storage of graphics commands, data, and textures. In at least one embodiment, the MCH 1016 may directly transmit data signals between the processor 1002, memory 1020, and other components in the computer system 1000, and bridge data signals between the processor bus 1010, memory 1020, and system I / O 1022. In at least one embodiment, the system logic chip may provide a graphics port for coupling to a graphics controller. In at least one embodiment, the MCH1016 may be coupled to memory 1020 through a high-bandwidth memory path 1018, and the graphics / video card 1012 may be coupled to the MCH1016 via an Accelerated Graphics Port ("AGP") interconnect 1014.

[0064] In at least one embodiment, the computer system 1000 may use a system I / O 1022, which is a proprietary hub interface bus for connecting the MCH 1016 to the I / O controller hub ("ICH") 1030. In at least one embodiment, the ICH 1030 may provide direct connectivity to several I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus may include, but is not limited to, a high-speed I / O bus for connecting peripherals to memory 1020, a chipset, and a processor 1002. Examples may include, but are not limited to, an audio controller 1029, a firmware hub ("Flash BIOS") 1028, a wireless transceiver 1026, data storage 1024, a legacy I / O controller 1023 including a user input and keyboard interface 1025, a serial expansion port 1027 such as a Universal Serial Bus ("USB"), and a network controller 1034. The data storage 1024 may comprise a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0065] In at least one embodiment, Figure 10 shows a system including interconnected hardware devices or “chips,” while in other embodiments, Figure 10 may show an exemplary system-on-a-chip (“SoC”). In at least one embodiment, devices may be interconnected by proprietary interconnects, standard interconnects (e.g., PCIe), or any combination thereof. In at least one embodiment, one or more components of computer system 1000 are interconnected using compute express link (CXL) interconnects.

[0066] The inference and / or training logic 815 is used to perform inference and / or training operations associated with one or more embodiments. Details relating to the inference and / or training logic 815 are provided below in conjunction with Figures 8a and / or 8b. In at least one embodiment, the inference and / or training logic 815 may be used in the system of Figure 10 for inference or prediction operations, at least in part, based on weight parameters calculated using the neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0067] Such components can be used to enable custom tasks to be performed using a model, such as a large-scale language model, without the need to retrain the same model.

[0068] Figure 11 is a block diagram showing an electronic device 1100 for utilizing a processor 1110 according to at least one embodiment. In at least one embodiment, the electronic device 1100 may be, for example, a notebook, a tower server, a rack server, a blade server, a laptop, a desktop, a tablet, a mobile device, a telephone, an embedded computer, or any other suitable electronic device.

[0069] In at least one embodiment, system 1100 may include a processor 1110 communicably coupled to any number or type of preferred components, peripherals, modules, or devices, but not limited to these. In at least one embodiment, the processor 1110 is coupled using a bus or interface such as a 1°C bus, a System Management Bus ("SMBus"), a Low Pin Count (LPC) bus, a Serial Peripheral Interface ("SPI"), a High Definition Audio ("HDA") bus, a Serial Advance Technology Attachment ("SATA") bus, a Universal Serial Bus ("USB") (versions 1, 2, or 3), or a Universal Asynchronous Receiver / Transmitter ("UART") bus. In at least one embodiment, Figure 11 shows a system including interconnected hardware devices or “chips,” while in other embodiments, Figure 11 may show an exemplary system-on-a-chip (“SoC”). In at least one embodiment, the devices shown in Figure 11 may be interconnected by proprietary interconnects, standard interconnects (e.g., PCIe), or any combination thereof. In at least one embodiment, one or more components of Figure 11 are interconnected using a Compute Express Link (CXL) interconnect.

[0070] In at least one embodiment, Figure 11 includes a display 1124, a touchscreen 1125, a touchpad 1130, a Near Field Communication ("NFC") unit 1145, a sensor hub 1140, a thermal sensor 1146, an Express Chipset ("EC") 1135, a Trusted Platform Module ("TPM") 1138, a BIOS / firmware / flash memory ("BIOS,FW flash") 1122, a DSP 1160, a drive 1120 such as a Solid State Disk ("SSD") or Hard Disk Drive ("HDD"), a Wireless Local Area Network ("WLAN") unit 1150, a Bluetooth unit 1152, and a Wireless Wide Area Network ("WWAN") unit. The components may include a network (1156), a Global Positioning System (GPS) (1155), a camera such as a USB 3.0 camera ("USB 3.0 camera") (1154), and / or a Low Power Double Data Rate ("LPDDR") memory unit ("LPDDR3") (1115) implemented, for example, in the LPDDR3 standard. Each of these components may be implemented in any preferred manner.

[0071] In at least one embodiment, other components may be communicatively coupled to the processor 1110 through the components described above. In at least one embodiment, the accelerometer 1141, the ambient light sensor ("ALS") 1142, the compass 1143, and the gyroscope 1144 may be communicatively coupled to the sensor hub 1140. In at least one embodiment, the thermal sensor 1139, the fan 1137, the keyboard 1146, and the touchpad 1130 may be communicatively coupled to the EC 1135. In at least one embodiment, the speaker 1163, the headphones 1164, and the microphone ("mic") 1165 may be communicatively coupled to the audio unit ("audio codec and class d amplifier") 1162, and the audio unit 1162 may be communicatively coupled to the DSP 1160. In at least one embodiment, the audio unit 1164 may include, for example, an audio coder / decoder ("codec") and a Class D amplifier. In at least one embodiment, a SIM card ("SIM") 1157 may be communicatively coupled to the WWAN unit 1156. In at least one embodiment, components such as the WLAN unit 1150 and the Bluetooth unit 1152, as well as the WWAN unit 1156, may be implemented in a Next Generation Form Factor ("NGFF").

[0072] The inference and / or training logic 815 is used to perform inference and / or training operations associated with one or more embodiments. Details relating to the inference and / or training logic 815 are provided below in conjunction with Figures 8a and / or 8b. In at least one embodiment, the inference and / or training logic 815 may be used in the system of Figure 10 for inference or prediction operations, at least in part, based on weight parameters calculated using the neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0073] Such components can be used to enable custom tasks to be performed using a model, such as a large-scale language model, without the need to retrain the same model.

[0074] Figure 12 is a block diagram of a processing system according to at least one embodiment. In at least one embodiment, system 1200 includes one or more processors 1202 and one or more graphics processors 1208, and may be a single-processor desktop system, a multi-processor workstation system, or a server system having a large number of processors 1202 or processor cores 1207. In at least one embodiment, system 1200 is a processing platform embedded in a system-on-a-chip (SoC) integrated circuit for use in a mobile device, handheld device, or embedded device.

[0075] In at least one embodiment, system 1200 may include, or may be incorporated within, a server-based gaming platform, a game console including a game and media console, a mobile gaming console, a handheld game console, or an online game console. In at least one embodiment, system 1200 is a mobile phone, a smartphone, a tablet computing device, or a mobile internet device. In at least one embodiment, processing system 1200 may also include, can be coupled with, or may be incorporated within, wearable devices such as a smartwatch wearable device, a smart eyewear device, an augmented reality device, or a virtual reality device. In at least one embodiment, processing system 1200 is a television or set-top box device having one or more processors 1202 and a graphical interface produced by one or more graphics processors 1208.

[0076] In at least one embodiment, one or more processors 1202 each include one or more processor cores 1207 for processing instructions that perform actions for system and user software when executed. In at least one embodiment, each of the one or more processor cores 1207 is configured to process a particular instruction set 1209. In at least one embodiment, the instruction set 1209 may facilitate computing via Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or Very Long Instruction Words (VLIW). In at least one embodiment, each processor core 1207 may process a different instruction set 1209, and the instruction set 1209 may include instructions to facilitate emulation of other instruction sets. In at least one embodiment, the processor core 1207 may also include other processing devices, such as a digital signal processor (DSP).

[0077] In at least one embodiment, the processor 1202 includes cache memory 1204. In at least one embodiment, the processor 1202 may have a single internal cache or multiple levels of internal caches. In at least one embodiment, the cache memory is shared among the various components of the processor 1202. In at least one embodiment, the processor 1202 also uses an external cache (e.g., a Level 3 (L3) cache or a Last Level Cache (LLC)) (not shown), and the external cache may be shared among the processor cores 1207 using known cache coherency techniques. In at least one embodiment, additionally, a register file 1206 is included in the processor 1202, and the register file 1206 may contain different types of registers for storing different types of data (e.g., integer registers, floating-point registers, status registers, and instruction pointer registers). In at least one embodiment, the register file 1206 may contain general-purpose registers or other registers.

[0078] In at least one embodiment, one or more processors 1202 are coupled with one or more interface buses 1210 to transmit communication signals, such as addresses, data, or control signals, between the processors 1202 and other components in the system 1200. In at least one embodiment, the interface bus 1210 in one embodiment may be a processor bus, such as a version of the Direct Media Interface (DMI) bus. In at least one embodiment, the interface 1210 is not limited to the DMI bus and may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), a memory bus, or other types of interface buses. In at least one embodiment, the (one or more) processors 1202 include an integrated memory controller 1216 and a platform controller hub 1230. In at least one embodiment, the memory controller 1216 facilitates communication between the memory device and other components of the system 1200, and the platform controller hub (PCH) 1230 provides connectivity to I / O devices via the local I / O bus.

[0079] In at least one embodiment, the memory device 1220 may be a dynamic random-access memory (DRAM) device, a static random-access memory (SRAM) device, a flash memory device, a phase-change memory device, or any other memory device having performance suitable for acting as process memory. In at least one embodiment, the memory device 1220 may act as system memory for the system 1200 to store data 1222 and instructions 1221 for use when one or more processors 1202 execute an application or process. In at least one embodiment, the memory controller 1216 may also be coupled to an optional external graphics processor 1212, which may communicate with one or more graphics processors 1208 in the processor 1202 to perform graphics and media operations. In at least one embodiment, the display device 1211 may be connected to one or more processors 1202. In at least one embodiment, the display device 1211 may include one or more internal display devices, such as those found in mobile electronic devices or laptop devices, or external display devices attached via a display interface (e.g., DisplayPort). In at least one embodiment, the display device 1211 may include a head-mounted display (HMD), such as a stereoscopic display device for use in virtual reality (VR) or augmented reality (AR) applications.

[0080] In at least one embodiment, the platform controller hub 1230 enables peripherals to connect to the memory device 1220 and processor 1202 via a high-speed I / O bus. In at least one embodiment, the I / O peripherals include, but are not limited to, an audio controller 1246, a network controller 1234, a firmware interface 1228, a wireless transceiver 1226, a touch sensor 1225, and a data storage device 1224 (e.g., a hard disk drive, flash memory, etc.). In at least one embodiment, the data storage device 1224 may be connected via a storage interface (e.g., SATA) or via a peripheral bus such as a peripheral component interconnect bus (e.g., PCI, PCI Express). In at least one embodiment, the touch sensor 1225 may include a touch screen sensor, a pressure sensor, or a fingerprint sensor. In at least one embodiment, the wireless transceiver 1226 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, or Long Term Evolution (LTE) transceiver. In at least one embodiment, the firmware interface 1228 enables communication with system firmware and may be, for example, a Unified Extensible Firmware Interface (UEFI). In at least one embodiment, the network controller 1234 can enable network connectivity to a wired network. In at least one embodiment, a high-performance network controller (not shown) is coupled to the interface bus 1210. In at least one embodiment, the audio controller 1246 is a multi-channel high-definition audio controller.In at least one embodiment, system 1200 includes an optional legacy I / O controller 1240 for connecting legacy devices (e.g., Personal System 2 (PS / 2)) to the system. In at least one embodiment, platform controller hub 1230 can also connect to one or more Universal Serial Bus (USB) controller 1242-connected input devices, such as a keyboard and mouse combination 1243, a camera 1244, or other USB input devices.

[0081] In at least one embodiment, instances of the memory controller 1216 and the platform controller hub 1230 may be integrated into a discreet external graphics processor, such as an external graphics processor 1212. In at least one embodiment, the platform controller hub 1230 and / or the memory controller 1216 may be external to one or more processors 1202. For example, in at least one embodiment, system 1200 may include an external memory controller 1216 and a platform controller hub 1230, which may be configured as a memory controller hub and peripheral controller hub within a system chipset communicating with (one or more) processors 1202.

[0082] The inference and / or training logic 815 is used to perform the inference and / or training operations associated with one or more embodiments. Details relating to the inference and / or training logic 815 are provided below in conjunction with Figures 8a and / or 8b. In at least one embodiment, part or all of the inference and / or training logic 815 may be incorporated into the graphics processor 1500. For example, in at least one embodiment, the training and / or inference techniques described herein may use one or more ALUs embodied in the graphics processor. Furthermore, in at least one embodiment, the inference and / or training operations described herein may be performed using logic other than the logic shown in Figure 8A or Figure 8B. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown) that constitute the ALUs of the graphics processor for performing one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0083] Such components can be used to enable custom tasks to be performed using a model, such as a large-scale language model, without the need to retrain the same model.

[0084] Figure 13 is a block diagram of a processor 1300 having one or more processor cores 1302A-1302N, an integrated memory controller 1314, and an integrated graphics processor 1308, according to at least one embodiment. In at least one embodiment, the processor 1300 may include additional cores up to additional core 1302N, represented by dashed boxes. In at least one embodiment, each of the processor cores 1302A-1302N includes one or more internal cache units 1304A-1304N. In at least one embodiment, each processor core also has access to one or more shared cached units 1306.

[0085] In at least one embodiment, the internal cache units 1304A-1304N and the shared cache unit 1306 represent a cache memory hierarchy within the processor 1300. In at least one embodiment, the cache memory units 1304A-1304N may include at least one level of instruction and data cache within each processor core, and one or more levels of shared intermediate level caches such as Level 2 (L2), Level 3 (L3), Level 4 (L4), or other levels of cache, where the highest level cache before external memory is classified as LLC. In at least one embodiment, cache coherency logic maintains coherency among the various cache units 1306 and 1304A-1304N.

[0086] In at least one embodiment, the processor 1300 may also include a set of one or more bus controller units 1316 and a system agent core 1310. In at least one embodiment, one or more bus controller units 1316 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, the system agent core 1310 provides management functionality for various processor components. In at least one embodiment, the system agent core 1310 includes one or more integrated memory controllers 1314 for managing access to various external memory devices (not shown).

[0087] In at least one embodiment, one or more of the processor cores 1302A to 1302N include support for simultaneous multithreading. In at least one embodiment, the system agent core 1310 includes components for coordinating and operating the cores 1302A to 1302N during multithreaded processing. In at least one embodiment, the system agent core 1310 may additionally include a power control unit (PCU), the PCU including logic and components for regulating the power state of one or more of the processor cores 1302A to 1302N and the graphics processor 1308.

[0088] In at least one embodiment, the processor 1300 additionally includes a graphics processor 1308 for performing graphics processing operations. In at least one embodiment, the graphics processor 1308 is coupled with a system agent core 1310 which includes a shared cache unit 1306 and one or more integrated memory controllers 1314. In at least one embodiment, the system agent core 1310 also includes a display controller 1311 for driving graphics processor outputs to one or more coupled displays. In at least one embodiment, the display controller 1311 may also be a separate module coupled with the graphics processor 1308 via at least one interconnection, or it may be incorporated within the graphics processor 1308.

[0089] In at least one embodiment, a ring-based interconnect unit 1312 is used to connect the internal components of the processor 1300. In at least one embodiment, alternative interconnect units such as point-to-point interconnects, switching interconnects, or other techniques may be used. In at least one embodiment, the graphics processor 1308 is connected to the ring interconnect 1312 via an I / O link 1313.

[0090] In at least one embodiment, I / O link 1313 represents at least one of several types of I / O interconnects, including on-package I / O interconnects that facilitate communication between various processor components and high-performance embedded memory modules 1318, such as eDRAM modules. In at least one embodiment, each of the processor cores 1302A to 1302N and the graphics processor 1308 use the embedded memory module 1318 as a shared last-level cache.

[0091] In at least one embodiment, the processor cores 1302A to 1302N are homogeneous cores that execute a common instruction set architecture. In at least one embodiment, the processor cores 1302A to 1302N are heterogeneous in terms of instruction set architecture (ISA), where one or more of the processor cores 1302A to 1302N execute a common instruction set, and the other cores of one or more of the processor cores 1302A to 1302N execute a subset of the common instruction set or a different instruction set. In at least one embodiment, the processor cores 1302A to 1302N are heterogeneous in terms of microarchitecture, where one or more cores with relatively high power consumption are coupled with one or more power cores with lower power consumption. In at least one embodiment, the processor 1300 may be implemented on one or more chips or as an SoC integrated circuit.

[0092] The inference and / or training logic 815 is used to perform the inference and / or training operations associated with one or more embodiments. Details relating to the inference and / or training logic 815 are provided below in conjunction with Figures 8a and / or 8b. In at least one embodiment, part or all of the inference and / or training logic 815 may be incorporated into the processor 1300. For example, in at least one embodiment, the training and / or inference techniques described herein may use one or more ALUs embodied in the graphics processor 1312, (one or more) graphics cores 1302A-1302N, or other components in Figure 13. Furthermore, in at least one embodiment, the inference and / or training operations described herein may be performed using logic other than the logic shown in Figure 8A or Figure 8B. In at least one embodiment, the weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown) that constitute the ALU of the graphics processor 1300 for implementing one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0093] Such components can be used to enable custom tasks to be performed using a model, such as a large-scale language model, without the need to retrain the same model.

[0094] Virtualized computing platform Figure 14 is an exemplary data flow diagram for a process 1400 that generates and deploys an image processing and inference pipeline, according to at least one embodiment. In at least one embodiment, process 1400 may be deployed in one or more facilities 1402 for use with imaging devices, processing devices, and / or other device types. Process 1400 may run within a training system 1404 and / or deployment system 1406. In at least one embodiment, training system 1404 may be used to carry out the training, deployment, and implementation of machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for use in deployment system 1406. In at least one embodiment, deployment system 1406 may be configured to offload processing and compute resources between distributed computing environments to reduce infrastructure requirements at facility 1402. In at least one embodiment, one or more applications in the pipeline may use or call services of the deployment system 1406 (e.g., inference, visualization, compute, AI, etc.) while the application is running.

[0095] In at least one embodiment, some of the applications used in the advanced processing and inference pipeline may use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, the machine learning model may be trained at facility 1402 using data 1408 (such as imaging data) generated at facility 1402 (and stored in one or more picture archiving and communication system (PACS) servers at facility 1402), or it may be trained using imaging or sequencing data 1408 from one or more other facilities, or a combination thereof. In at least one embodiment, the training system 1404 may be used to provide applications, services, and / or other resources for generating a working and deployable machine learning model for the deployment system 1406.

[0096] In at least one embodiment, the model registry 1424 may be backed up by object storage capable of supporting version control and object metadata. In at least one embodiment, the object storage may be accessible, for example, from within a cloud platform, through a cloud storage (e.g., cloud 1226 in Figure 12) compatible application programming interface (API). In at least one embodiment, machine learning models in the model registry 1424 may be uploaded, listed, modified, or deleted by a system developer or partner interacting with the API. In at least one embodiment, the API may provide access to a way that enables appropriately qualified users to associate models with applications, thereby allowing the models to run as part of a containerized instantiation of the application.

[0097] In at least one embodiment, the training pipeline 1404 (Figure 14) may include a scenario in which the facility 1402 is training its own machine learning model or has an existing machine learning model that needs to be optimized or updated. In at least one embodiment, imaging data 1408 generated by (one or more) imaging devices, sequencing devices, and / or other device types may be received. In at least one embodiment, once the imaging data 1408 is received, AI-assisted annotation 1410 may be used to help generate annotations corresponding to the imaging data 1408 to be used as ground truth data for the machine learning model. In at least one embodiment, AI-assisted annotation 1410 may include one or more machine learning models (e.g., convolutional neural networks (CNNs)), and one or more machine learning models may be trained to generate annotations corresponding to several types of imaging data 1408 (e.g., from several devices). In at least one embodiment, the AI-assisted annotation 1410 may then be used directly to generate ground truth data, or it may be modified or fine-tuned using an annotation tool. In at least one embodiment, the AI-assisted annotation 1410, labeled clinic data 1412, or a combination thereof may be used as ground truth data for training a machine learning model. In at least one embodiment, the trained machine learning model may be referred to as output model 1416 and may be used by the deployment system 1406 as described herein.

[0098] In at least one embodiment, the training pipeline 1204 (Figure 12) may include a scenario in which facility 1402 requires a machine learning model to use when performing one or more processing tasks for one or more applications in the deployment system 1406, but facility 1402 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, an existing machine learning model may be selected from the model registry 1424. In at least one embodiment, the model registry 1424 may include machine learning models trained to perform a variety of different inference tasks on imaging data. In at least one embodiment, the machine learning models in the model registry 1424 may be trained on imaging data from a different facility than facility 1402 (e.g., a facility located remotely). In at least one embodiment, the machine learning model may be trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when trained on imaging data from a specific location, training may be performed at that location, or at least in a manner that protects the confidentiality of the imaging data or restricts the transfer of the imaging data outside the premises. In at least one embodiment, once a model is trained, or partially trained, at one location, the machine learning model may be added to the model registry 1424. In at least one embodiment, the machine learning model may then be retrained or updated at any number of other locations, and the retrained or updated model may be made available in the model registry 1424. In at least one embodiment, a machine learning model may then be selected from the model registry 1424, which may be referred to as an output model 1416, and may be used in the deployment system 1406 to perform one or more processing tasks for one or more applications of the deployment system.

[0099] In at least one embodiment, the training pipeline 1204 (Figure 12), scenario may include a facility 1402 requiring a machine learning model for use in performing one or more processing tasks for one or more applications in the deployment system 1406, but facility 1402 may not currently have such a machine learning model (or may not have an efficient or effective model optimized for such purposes). In at least one embodiment, a machine learning model selected from the model registry 1424 may not be fine-tuned or optimized for imaging data 1408 generated at facility 1402 due to differences in the population, the robustness of the training data used to train the machine learning model, the diversity of anomalies in the training data, and / or other issues associated with the training data. In at least one embodiment, AI-assisted annotation 1410 may be used to help generate annotations corresponding to the imaging data 1408 to be used as ground truth data for retraining or updating the machine learning model. In at least one embodiment, labeled data 1412 may be used as ground truth data for training the machine learning model. In at least one embodiment, retraining or updating a machine learning model may be referred to as model training 1414. In at least one embodiment, model training 1414, for example, AI-assisted annotation 1410, labeled clinic data 1412, or a combination thereof, may be used as ground truth data for retraining or updating a machine learning model. In at least one embodiment, the trained machine learning model may be referred to as output model 1416, which may be used by the deployment system 1406 as described herein.

[0100] In at least one embodiment, the deployment system 1406 may include software 1418, services 1420, hardware 1422, and / or other components, features, and functionalities. In at least one embodiment, the deployment system 1406 may include a software "stack" so that software 1418 can be built on top of services 1420 and use services 1420 to perform some or all of the processing tasks, and services 1420 and software 1418 can be built on top of hardware 1422 and use hardware 1422 to perform the processing, storage, and / or other compute tasks of the deployment system 1406. In at least one embodiment, software 1418 may include any number of different containers, each of which may perform the instantiation of an application. In at least one embodiment, each application may perform one or more processing tasks in the advanced processing and inference pipeline (e.g., inference, object detection, feature detection, segmentation, image enhancement, calibration, etc.). In at least one embodiment, the advanced processing and inference pipeline may be defined based on the selection of different containers desired or required to process the imaging data 1408, in addition to a container that receives and configures the imaging data for use by each container and / or by facility 1402 after processing it through the pipeline (for example, to convert the output to an available data type). In at least one embodiment, the combination of containers in software 1418 (for example, building the pipeline) may be referred to as a virtual device (as described in more detail herein), and the virtual device may leverage services 1420 and hardware 1422 to perform some or all of the processing tasks of applications instantiated in the containers.

[0101] In at least one embodiment, the data processing pipeline may receive input data (e.g., imaging data 1408) in a specific format in response to an inference request (e.g., a request from a user of the deployment system 1406). In at least one embodiment, the input data may represent one or more images, videos, and / or other data representations generated by one or more imaging devices. In at least one embodiment, the data may undergo preprocessing as part of the data processing pipeline to prepare the data for processing by one or more applications. In at least one embodiment, postprocessing may be performed on the output of one or more inference tasks or other processing tasks in the pipeline to prepare output data for subsequent applications and / or output data for user transmission and / or use (e.g., as a response to an inference request). In at least one embodiment, the inference task may be performed by one or more machine learning models, such as a trained or deployed neural network, which may include the output model 1416 of the training system 1404.

[0102] In at least one embodiment, tasks in a data processing pipeline may be encapsulated in one or more containers, each representing a separate, fully functional instantiation of an application and a virtualized computing environment capable of referencing machine learning models. In at least one embodiment, a container or application may be exposed to a private (e.g., restricted access) area of ​​a container registry (described in more detail herein), and trained or deployed models may be stored in a model registry 1424 and associated with one or more applications. In at least one embodiment, an image of an application (e.g., a container image) may be available in the container registry, and once selected by a user from the container registry for deployment in a pipeline, the image may be used to generate a container for instantiation of the application for use by the user's system.

[0103] In at least one embodiment, a developer (e.g., a software developer, clinician, physician, etc.) may develop, publish, and store an application (e.g., as a container) for performing image processing and / or inference on supplied data. In at least one embodiment, development, publication, and / or storage may be carried out using a software development kit (SDK) associated with the system (e.g., to ensure that the developed application and / or container conforms to or is compatible with the system). In at least one embodiment, the developed application may be tested locally (e.g., at a first facility, against data from a first facility) using an SDK that can support at least some of the services 1420 as a system (e.g., system 1200 in Figure 12). In at least one embodiment, because a DICOM object may contain anywhere one to hundreds of images or other data types, and due to data variability, the developer may be responsible for managing the extraction and preparation of incoming data (e.g., setting up constructs for the application, incorporating preprocessing into the application, etc.). In at least one embodiment, once validated by System 1400 (for example, with respect to accuracy), the application may become available in a container registry for user selection and / or implementation to perform one or more processing tasks on data at the user's facility (for example, a second facility).

[0104] In at least one embodiment, the developer may then share the application or container over a network for access and use by users of the system (e.g., system 1400 in Figure 14). In at least one embodiment, the completed and validated application or container may be stored in a container registry, and the associated machine learning models may be stored in a model registry 1424. In at least one embodiment, a requesting entity providing an inference or image processing request may browse the container registry and / or model registry 1424 for applications, containers, datasets, machine learning models, etc., select a desired combination of elements to include in the data processing pipeline, and submit an imaging processing request. In at least one embodiment, the request may include input data (and, in some examples, associated patient data) necessary to perform the request, and / or include the selection of (one or more) applications and / or machine learning models to be executed when processing the request. In at least one embodiment, the request may then be passed to one or more components of the deployment system 1406 (e.g., the cloud) to perform the processing in the data processing pipeline. In at least one embodiment, processing by the deployment system 1406 may include referencing selected elements (e.g., applications, containers, models, etc.) from the container registry and / or model registry 1424. In at least one embodiment, once the pipeline has generated results, the results may be returned to the user for reference (e.g., for viewing in a viewing application suite running on a local, on-premises workstation or terminal).

[0105] In at least one embodiment, service 1420 may be utilized to assist in the processing or execution of an application or container in a pipeline. In at least one embodiment, service 1420 may include compute services, artificial intelligence (AI) services, visualization services, and / or other service types. In at least one embodiment, service 1420 may provide functionality common to one or more applications in software 1418, and thus the functionality may be abstracted to a service that can be called or utilized by the applications. In at least one embodiment, the functionality provided by service 1420 may operate dynamically and more efficiently, and may scale well by enabling applications to process data in parallel (for example, using the parallel computing platform 1230 (Figure 12)). In at least one embodiment, service 1420 may be shared among and between various applications, rather than each application sharing the same functionality provided by service 1420 being required to have its own instance of service 1420. In at least one embodiment, the service may include, in non-limiting examples, an inference server or engine that can be used to perform detection or segmentation tasks. In at least one embodiment, a model training service may be included that can provide machine learning model training and / or retraining capabilities. In at least one embodiment, a data augmentation service may further be included that can provide GPU-accelerated data (e.g., DICOM, RIS, CIS, REST-compliant, RPC, raw, etc.) extraction, resizing, scaling, and / or other augmentation. In at least one embodiment, a visualization service may be used that can add image rendering effects such as ray tracing, rasterization, denoising, and sharpening to add realism to two-dimensional (2D) and / or three-dimensional (3D) models.In at least one embodiment, a virtual instrument service may be included that provides beamforming, segmentation, inference, imaging, and / or support for other applications within the virtual instrument pipeline.

[0106] In at least one embodiment, if service 1420 includes an AI service (e.g., an inference service), one or more machine learning models may be executed by calling the inference service (e.g., an inference server) (e.g., as an API call) to execute the machine learning models, or their processing, as part of the application execution. In at least one embodiment, if another application includes one or more machine learning models for a segmentation task, the application may call the inference service to execute the machine learning models to perform one or more processing operations associated with the segmentation task. In at least one embodiment, software 1418 implementing an advanced processing and inference pipeline including a segmentation application and an anomaly detection application may be streamlined because each application may call the same inference service to perform one or more inference tasks.

[0107] In at least one embodiment, the hardware 1422 may include a GPU, CPU, graphics card, AI / deep learning system (e.g., an AI supercomputer such as NVIDIA's DGX), cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 1422 may be used to provide efficient and dedicated support for the software 1418 and service 1420 in deployment system 1406. In at least one embodiment, the use of GPU processing for processing locally (e.g., at facility 1402) may be implemented within the AI / deep learning system, in the cloud system, and / or in other processing components of deployment system 1406 to improve the efficiency, accuracy, and effectiveness of image processing and generation. In at least one embodiment, the software 1418 and / or service 1420 may, as a non-limiting example, be optimized for GPU processing relating to deep learning, machine learning, and / or high-performance computing. In at least one embodiment, at least a portion of the computing environment of the deployment system 1406 and / or training system 1404 may be run on one or more supercomputers or high-performance computing systems in a data center using GPU-optimized software (e.g., a combination of hardware and software from NVIDIA's DGX system). In at least one embodiment, the hardware 1422 may include any number of GPUs, which may be called to perform parallel processing of data as described herein. In at least one embodiment, the cloud platform may further include GPU processing for GPU-optimized execution of deep learning tasks, machine learning tasks, or other computing tasks. In at least one embodiment, the cloud platform (e.g., NVIDIA's NGC) may be run using (one or more) AI / deep learning supercomputers and / or GPU-optimized software (e.g., provided on NVIDIA's DGX system) as a hardware abstraction and scaling platform.In at least one embodiment, the cloud platform may integrate an application container clustering system or orchestration system (e.g., Kubernetes) for multiple GPUs to enable seamless scaling and load balancing.

[0108] Figure 15 is a system diagram for an exemplary system 1500 for generating and implementing an imaging introduction pipeline, according to at least one embodiment. In at least one embodiment, system 1500 may be used to implement process 1400 of Figure 14, and / or other processes including advanced processing and inference pipelines. In at least one embodiment, system 1500 may include a training system 1404 and an introduction system 1406. In at least one embodiment, the training system 1404 and the introduction system 1406 may be implemented using software 1418, services 1420, and / or hardware 1422 as described herein.

[0109] In at least one embodiment, System 1500 (e.g., training system 1404 and / or deployment system 1406) may be implemented in a cloud computing environment (e.g., using cloud 1526). In at least one embodiment, System 1500 may be implemented locally with respect to a healthcare service facility, or as a combination of both cloud computing resources and local computing resources. In at least one embodiment, access to APIs in cloud 1526 may be restricted to authorized users through established security measures or protocols. In at least one embodiment, the security protocol may include web tokens, which may be signed by an authentication service (e.g., AuthN, AuthZ, Gluecon, etc.) and may have appropriate authorizations. In at least one embodiment, APIs of virtual devices (as described herein), or other instantiations of System 1500, may be restricted to a set of public IPs that are inspected or authorized for interaction.

[0110] In at least one embodiment, various components of System 1500 may communicate with one another using any of various different network types, including, but not limited to, local area networks (LANs) and / or wide area networks (WANs), via wired and / or wireless communication protocols. In at least one embodiment, communication between a facility and components of System 1500 (e.g., for sending inference requests, for receiving the results of inference requests) may be conducted via (one or more) data buses, wireless data protocols (Wi-Fi), wired data protocols (e.g., Ethernet), etc.

[0111] In at least one embodiment, the training system 1404 may execute a training pipeline 1504 similar to that described herein with respect to Figure 14. In at least one embodiment, if one or more machine learning models are to be used in the introduction pipeline 1510 by the introduction system 1406, the training pipeline 1504 may be used to train or retrain one or more (e.g., pre-trained) models, and / or to implement one or more of the pre-trained models 1506 (e.g., without the need for retraining or updating). In at least one embodiment, one or more output models 1416 may be produced as a result of the training pipeline 1504. In at least one embodiment, the training pipeline 1504 may include any number of processing steps, including, but not limited to, image data (or other input data) conversion or adaptation. In at least one embodiment, different training pipelines 1504 may be used for different machine learning models used by the introduction system 1406. In at least one embodiment, a training pipeline 1504 similar to the first example described with respect to Figure 14 may be used for a first machine learning model, a training pipeline 1504 similar to the second example described with respect to Figure 14 may be used for a second machine learning model, and a training pipeline 1504 similar to the third example described with respect to Figure 14 may be used for a third machine learning model. In at least one embodiment, any combination of tasks within the training system 1404 may be used depending on what is required for each respective machine learning model. In at least one embodiment, one or more of the machine learning models may already be trained and ready for deployment, and therefore the machine learning models may not undergo any processing by the training system 1404 and may be implemented by the deployment system 1406.

[0112] In at least one embodiment, the (one or more) output models 1416 and / or the (one or more) pre-trained models 1506 may include any type of machine learning model depending on the implementation or embodiment. In at least one embodiment, and not limited to, the machine learning models used by System 1500 may include (one or more) machine learning models using linear regression, logistic regression, decision trees, support vector machines (SVMs), Naive Bayes, k-nearest neighbors (Knn), k-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutions, recurrents, perceptrons, Long / Short Term Memory (LSTMs), Hopfield, Boltzmann, deep belief, deconvolution, adversarial generative, liquid state machines, etc.), and / or other types of machine learning models.

[0113] In at least one embodiment, the training pipeline 1504 may include AI-assisted annotation, as will be described in more detail herein with respect to at least Figure 15B. In at least one embodiment, labeled data 1412 (e.g., conventional annotation) may be generated by any number of techniques. In at least one embodiment, labels or other annotations may be generated in a drawing program (e.g., an annotation program), a computer-aided design (CAD) program, a labeling program, another type of program suitable for generating annotations or labels for ground truth, and / or, in some examples, by handwriting. In at least one embodiment, the ground truth data may be synthetically generated (e.g., generated from a computer model or rendering), realistically generated (e.g., designed and generated from real-world data), machine-automated (e.g., using feature analysis and learning to extract features from the data and then generate labels), human-annotated (e.g., a labeller or annotation expert defines the location of the labels), and / or a combination thereof. In at least one embodiment, for each instance of the imaging data 1408 (or other data types used by the machine learning model), there may be corresponding ground truth data generated by the training system 1404. In at least one embodiment, AI-assisted annotation may be performed as part of the introduction pipeline 1510, either in addition to or instead of AI-assisted annotation included in the training pipeline 1504. In at least one embodiment, the system 1500 may include a multilayer platform, which may include a software layer (e.g., software 1418) of a diagnostic application (or other application type) capable of performing one or more medical imaging and diagnostic functions. In at least one embodiment, the system 1500 may be communicably coupled (e.g., via an encrypted link) to a PACS server network of one or more facilities.In at least one embodiment, the system 1500 may be configured to access and reference data from a PACS server in order to perform operations such as training a machine learning model, deploying a machine learning model, image processing, inference, and / or other operations.

[0114] In at least one embodiment, the software layer may be implemented as a secure, encrypted, and / or authenticated API through which an application or container may be invoked (e.g., called) from (one or more) external environments (e.g., facility 1402). In at least one embodiment, the application may then call or execute one or more services 1420 to perform compute, AI, or visualization tasks associated with each application, and the software 1418 and / or services 1420 may leverage the hardware 1422 to perform the processing tasks in an effective and efficient manner.

[0115] In at least one embodiment, the introduction system 1406 may execute an introduction pipeline 1510. In at least one embodiment, the introduction pipeline 1510 may include any number of applications that may be applied sequentially, discontinuously, or otherwise to imaging data (and / or other data types) generated by imaging devices, sequencing devices, genomics devices, etc., including AI-assisted annotation as described above. In at least one embodiment, as described herein, the introduction pipeline 1510 for individual devices may be referred to as a virtual instrument for the device (e.g., a virtual ultrasound instrument, a virtual CT scanning instrument, a virtual sequencing instrument, etc.). In at least one embodiment, there may be two or more introduction pipelines 1510 for a single device, depending on the information desired from the data generated by the device. In at least one embodiment, there may be a first introduction pipeline 1510 if anomaly detection is desired from the MRI machine, and there may be a second introduction pipeline 1510 if image enhancement is desired from the output of the MRI machine.

[0116] In at least one embodiment, the image generation application may include processing tasks that involve the use of a machine learning model. In at least one embodiment, the user may wish to use their own machine learning model or select a machine learning model from the model registry 1424. In at least one embodiment, the user may implement their own machine learning model or select a machine learning model to include in the application in order to perform the processing tasks. In at least one embodiment, the application may be selectable and customizable, and by defining the construction of the application, the deployment and implementation of the application for a particular user is presented as a more seamless user experience. In at least one embodiment, by leveraging other features of the system 1500, such as services 1420 and hardware 1422, the deployment pipeline 1510 may become even more user-friendly, provide easier integration, and produce more accurate, efficient, and timely results.

[0117] In at least one embodiment, the deployment system 1406 may include a user interface 1514 (e.g., a graphical user interface, a web interface, etc.) which may be used to select applications for inclusion in one or more deployment pipelines 1510, to deploy applications, to modify or change applications or their parameters or construction, to use and interact with one or more deployment pipelines 1510 during setup and / or deployment, and / or to interact with the deployment system 1406 in other ways. In at least one embodiment, although not shown with respect to the training system 1404, the user interface 1514 (or a different user interface) may be used to select models for use in the deployment system 1406, to select models for training or retraining in the training system 1404, and / or to interact with the training system 1404 in other ways.

[0118] In at least one embodiment, a pipeline manager 1512 may be used in addition to the application orchestration system 1528 to manage interactions between applications or containers in (one or more) deployment pipelines 1510 and services 1420 and / or hardware 1422. In at least one embodiment, the pipeline manager 1512 may be configured to facilitate application-to-application interactions, applications-to-service interactions, and / or applications or services-to-hardware interactions. In at least one embodiment, it is shown to be included in software 1418, but this is not intended to limit it, and in some examples (for example, shown in Figure 12cc), the pipeline manager 1512 may be included in services 1420. In at least one embodiment, the application orchestration system 1528 (e.g., Kubernetes, DOCKER, etc.) may include a container orchestration system, which may group applications into containers as logical units for coordinating, managing, scaling, and deployment. In at least one embodiment, by associating applications (e.g., reconstruction applications, segmentation applications, etc.) from one or more deployment pipelines 1510 with individual containers, each application can run in a self-contained environment (e.g., at the kernel level) to improve speed and efficiency.

[0119] In at least one embodiment, each application and / or container (or its image) may be developed, modified, and deployed individually (for example, a first user or developer may develop, modify, and deploy a first application, and a second user or developer may develop, modify, and deploy a second application independently of the first user or developer), which may allow for focusing and paying attention to the tasks of a single application and / or container without being interrupted by the tasks of other applications or containers. In at least one embodiment, communication and coordination between different containers or applications may be assisted by the pipeline manager 1512 and the application orchestration system 1528. In at least one embodiment, the application orchestration system 1528 and / or the pipeline manager 1512 may facilitate communication between each of the applications or containers and the sharing of resources between them, as long as the expected inputs and / or outputs of each container or application are known by the system (for example, based on the construction of the application or container). In at least one embodiment, one or more applications or containers in the deployment pipeline 1510 may share the same services and resources, so that the application orchestration system 1528 can orchestrate, load balance, and decide on sharing services or resources among the various applications or containers. In at least one embodiment, a scheduler may be used to track the resource requirements of the applications or containers, the current or planned usage of these resources, and the availability of the resources. In at least one embodiment, the scheduler may therefore allocate resources to different applications and distribute resources among applications, taking into account the system requirements and availability.In some examples, the scheduler (and / or other components of the application orchestration system 1528) may determine the availability and distribution of resources based on constraints imposed on the system (e.g., user constraints), such as quality of service (QoS) and the urgency of the need for data output (e.g., to determine whether real-time processing or delayed processing should be performed).

[0120] In at least one embodiment, the services 1420 utilized and shared by applications or containers in the deployment system 1406 may include compute services 1516, AI services 1518, visualization services 1520, and / or other service types. In at least one embodiment, an application may call (e.g., execute) one or more of the services 1420 to perform processing operations for the application. In at least one embodiment, compute service 1516 may be utilized by an application to perform supercomputing or other high-performance computing (HPC) tasks. In at least one embodiment, one or more compute services 1516 may be utilized to process data through one or more applications and / or to perform parallel processing (e.g., using a parallel computing platform 1530) for substantially simultaneous processing of one or more tasks of a single application. In at least one embodiment, the parallel computing platform 1530 (e.g., NVIDIA's CUDA) may enable general-purpose computing on GPUs (GPGPU: GPGPU) on a GPU (e.g., GPU 1522). In at least one embodiment, the software layer of the parallel computing platform 1530 may provide access to a virtual instruction set and parallel computing elements of the GPU for the execution of a compute kernel. In at least one embodiment, the parallel computing platform 1530 may include memory, which in some embodiments may be shared among multiple containers and / or among different processing tasks within a single container.In at least one embodiment, inter-process communication (IPC) calls may be generated for multiple containers and / or multiple processes within containers to use the same data from a shared segment of memory on the parallel computing platform 1530 (for example, when multiple different stages of an application or multiple applications are processing the same information). In at least one embodiment, the same data in the same location in memory may be used for any number of processing tasks (for example, at the same time, at different times, etc.) rather than making copies of the data and moving the data to different locations in memory (e.g., read / write operations). In at least one embodiment, when data is used and new data is generated as a result of processing, this information about the new location of the data may be stored and shared among various applications. In at least one embodiment, the location of the data and the location of the updated or modified data may be part of the definition of how the payload is understood within the container.

[0121] In at least one embodiment, the AI ​​service 1518 may be leveraged to perform inference services for running one or more machine learning models associated with an application (e.g., assigned the role of performing one or more processing tasks of the application). In at least one embodiment, the AI ​​service 1518 may leverage the AI ​​system 1524 to run one or more machine learning models (e.g., neural networks such as CNNs) for segmentation, reconstruction, object detection, feature detection, classification, and / or other inference tasks. In at least one embodiment, an application in one or more introductory pipelines 1510 may perform inference on imaging data using one or more output models 1416 from the training system 1404 and / or other models of the application. In at least one embodiment, two or more examples of inference using the application orchestration system 1528 (e.g., a scheduler) may be available. In at least one embodiment, the first category may include high-priority / low-latency routes that can achieve a higher service level agreement, such as for performing reasoning regarding urgent requests in emergencies or for radiologists during diagnosis. In at least one embodiment, the second category may include standard-priority routes that may be used for requests that may not be urgent or where analysis may be performed later. In at least one embodiment, the application orchestration system 1528 may distribute resources (e.g., services 1420 and / or hardware 1422) based on priority routes for different reasoning tasks of the AI ​​service 1518.

[0122] In at least one embodiment, shared storage may be attached to the AI ​​service 1518 within system 1500. In at least one embodiment, shared storage may act as a cache (or other storage device type) and may be used to process inference requests from an application. In at least one embodiment, when an inference request is submitted, the request may be received by a set of API instances of deployment system 1406, and one or more instances may be selected to process the request (e.g., for best fit, for load balancing, etc.). In at least one embodiment, to process the request, the request may be placed in a database, a machine learning model may be located from the model registry 1424 if it is not already in the cache, and a verification step may ensure that a suitable machine learning model is loaded into the cache (e.g., shared storage) and / or a copy of the model can be stored in the cache. In at least one embodiment, if the application is not yet running or there are not enough instances of the application, a scheduler (e.g., of pipeline manager 1512) may be used to start the application referenced in the request. In at least one embodiment, an inference server may be started if one has not yet been started to run the model. Any number of inference servers may be started for each model. In at least one embodiment, in a clustered pull model, the model may be cached whenever load balancing is advantageous. In at least one embodiment, the inference server may be statically loaded onto the corresponding distributed server.

[0123] In at least one embodiment, inference may be performed using an inference server running in a container. In at least one embodiment, an instance of the inference server may be associated with a model (optionally, multiple versions of the model). In at least one embodiment, when a request to perform inference against a model is received and no instance of the inference server exists, a new instance may be loaded. In at least one embodiment, when the inference server is started, a model may be passed to the inference server, thereby allowing the same container to be used to serve different models, as long as the inference server is running as a different instance.

[0124] In at least one embodiment, while the application is running, an inference request for a given application may be received, a container (e.g., hosting an instance of the inference server) may be loaded (if not already loaded), and a start procedure may be called. In at least one embodiment, preprocessing logic in the container may load, decode, and / or perform any additional preprocessing on incoming data (e.g., using one or more CPUs and / or one or more GPUs). In at least one embodiment, once the data is prepared for inference, the container may perform inference on the data as needed. In at least one embodiment, this may involve a single inference call for a single image (e.g., an X-ray of a hand) or may require inference on hundreds of images (e.g., a CT scan of a chest). In at least one embodiment, the application may summarize results before completion, which may include, but are not limited to, generating a single confidence score, pixel-level segmentation, voxel-level segmentation, visualizations, or text to summarize the findings. In at least one embodiment, different models or applications may be assigned different priorities. For example, some models may have real-time priority (TAT < 1 minute), while others may have low priority (e.g., TAT < 10 minutes). In at least one embodiment, model execution time may be measured from the requesting agency or entity and may include partner network traversal time and execution for inference services.

[0125] In at least one embodiment, the transfer of requests between service 1420 and the inference application may be hidden behind a software development kit (SDK), and robust transport may be provided through a queue. In at least one embodiment, for each application / tenant ID combination, requests are queued via an API, and the SDK pulls requests from the queue and delivers them to the application. In at least one embodiment, a name for the queue may be provided in the environment in which the SDK picks up requests. In at least one embodiment, asynchronous communication through a queue may be useful because the communication may allow any instance of the application to pick up the work when the work becomes available. The results may be returned through the queue to ensure that no data is lost. In at least one embodiment, the queue may also provide the ability to segment work, so that the highest priority work may proceed to a queue to which most instances of the application are connected, while the lowest priority work may proceed to a queue to which a single instance is connected, processing tasks in the order they are received. In at least one embodiment, the application may run on a GPU-accelerated instance generated in cloud 1526, and the inference service may perform inference on the GPU.

[0126] In at least one embodiment, the visualization service 1520 may be utilized to generate visualizations for viewing the output of an application and / or (one or more) introductory pipelines 1510. In at least one embodiment, the visualization service 1520 may utilize the GPU 1522 to generate visualizations. In at least one embodiment, rendering effects such as ray tracing may be implemented by the visualization service 1520 to generate higher quality visualizations. In at least one embodiment, visualizations may include, but are not limited to, 2D image rendering, 3D volume rendering, 3D volume reconstruction, 2D tomographic slicing, virtual reality display, augmented reality display, etc. In at least one embodiment, a virtualized environment may be used to generate a virtual interactive display or environment (e.g., a virtual environment) for interaction by a user of the system (e.g., a doctor, nurse, radiologist, etc.). In at least one embodiment, the visualization service 1520 may include an internal visualizer, cinematics, and / or other rendering or image processing capabilities or functionalities (e.g., ray tracing, rasterization, internal optics, etc.).

[0127] In at least one embodiment, hardware 1422 may include GPU 1522, AI system 1524, cloud 1526, and / or any other hardware used to run training system 1404 and / or deployment system 1406. In at least one embodiment, GPU 1522 (e.g., NVIDIA's TESLA and / or QUADRO GPUs) may include any number of GPUs, any number of GPUs may be used to perform processing tasks for any of the compute service 1516, AI service 1518, visualization service 1520, other services, and / or features or functionalities of software 1418. For example, with respect to AI service 1518, GPU 1522 may be used to perform preprocessing on imaging data (or other data types used by the machine learning model), postprocessing on the output of the machine learning model, and / or perform inference (e.g., to run the machine learning model). In at least one embodiment, the cloud 1526, the AI ​​system 1524, and / or other components of system 1500 may use the GPU 1522. In at least one embodiment, the cloud 1526 may include a GPU-optimized platform for deep learning tasks. In at least one embodiment, the AI ​​system 1524 may use a GPU, and the cloud 1526, or at least a portion assigned the role of deep learning or inference, may run using one or more AI systems 1524. Thus, although hardware 1422 is shown as a separate component, this is not intended to be limiting, and any component of hardware 1422 may be combined with or leveraged by any other component of hardware 1422.

[0128] In at least one embodiment, the AI ​​system 1524 may include a dedicated computing system (e.g., a supercomputer or HPC) configured for inference, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, the AI ​​system 1524 (e.g., NVIDIA's DGX) may include GPU-optimized software (e.g., a software stack), which may run using multiple GPUs 1522 in addition to the CPU, RAM, storage, and / or other components, features, or functionalities. In at least one embodiment, one or more AI systems 1524 may be implemented in a cloud 1526 (e.g., in a data center) to perform some or all of the AI-based processing tasks of system 1500.

[0129] In at least one embodiment, cloud 1526 may include GPU acceleration infrastructure (e.g., NVIDIA's NGC), which may provide a GPU-optimized platform for performing processing tasks of system 1500. In at least one embodiment, cloud 1526 may include (one or more) AI systems 1524 (e.g., as a hardware abstraction and scaling platform) for performing one or more of the AI-based tasks of system 1500. In at least one embodiment, cloud 1526 may integrate with an application orchestration system 1528, leveraging multiple GPUs to enable seamless scaling and load balancing between applications and services 1420. In at least one embodiment, cloud 1526 may be tasked with performing at least some of the services 1420 of system 1500, including compute service 1516, AI service 1518, and / or visualization service 1520, as described herein. In at least one embodiment, Cloud 1526 may perform large and small batch inference (e.g., running NVIDIA's TENSOR RT), provide accelerated parallel computing APIs and platforms 1530 (e.g., NVIDIA's CUDA), run application orchestration system 1528 (e.g., KUBERNETES), provide graphics rendering APIs and platforms (e.g., for ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques to produce higher quality cinematics), and / or provide other functionality for system 1500.

[0130] Figure 15A shows a data flow diagram of process 1500 for training, retraining, or updating a machine learning model according to at least one embodiment. In at least one embodiment, process 1500 may be performed using system 1500 in Figure 15 as a non-limiting example. In at least one embodiment, process 1500 may leverage the services 1420 and / or hardware 1422 of system 1500 as described herein. In at least one embodiment, the improved model 1512 produced by process 1500 may be executed by deployment system 1406 for one or more containerized applications in deployment pipeline 1510.

[0131] In at least one embodiment, model training 1414 may include retraining or updating the initial model 1504 (e.g., a pre-trained model) using new training data (e.g., customer dataset 1506, and / or new input data such as new ground truth data associated with the input data). In at least one embodiment, in order to retrain or update the initial model 1504, one or more output or loss layers of the initial model 1504 may be reset or deleted and / or replaced with one or more updated or new output or loss layers. In at least one embodiment, the initial model 1504 may have parameters (e.g., weights and / or biases) that have been previously fine-tuned and remain from previous training, and therefore training or retraining 1414 may not take as long or require as much processing as training the model from scratch. In at least one embodiment, during model training 1414, by having one or more reset or replaced output or loss layers of the initial model 1504, the parameters may be updated and readjusted for the new dataset based on the loss calculation associated with the accuracy of one or more output or loss layers when generating predictions with respect to the new customer dataset 1506 (e.g., image data 1408 in Figure 14).

[0132] In at least one embodiment, the pre-trained model 1506 may be stored in a data store or registry (for example, the model registry 1424 in Figure 14). In at least one embodiment, the pre-trained model 1506 may be trained at least partially at one or more facilities other than the facility where process 1500 is performed. In at least one embodiment, in order to protect the privacy and rights of patients, subjects, or customers at different facilities, the pre-trained model 1506 may be trained on-site using on-site generated customer or patient data. In at least one embodiment, the pre-trained model 1506 may be trained using the cloud 1526 and / or other hardware 1422, but privacy-protected sensitive patient data may not be transferred to any component of the cloud 1526 (or other off-site hardware), may not be used by such components, or may be inaccessible to such components. In at least one embodiment, if the pre-trained model 1506 is trained using patient data from two or more facilities, the pre-trained model 1506 may be trained individually for each facility and then trained with respect to patient or customer data from another facility. In at least one embodiment, customer or patient data from any number of facilities may be used to train the pre-trained model 1506 on-premises and / or off-premises, such as in a data center or other cloud computing infrastructure, if the customer or patient data is released from privacy concerns (e.g., by waiver, for experimental use, etc.) or if the customer or patient data is included in a public data set.

[0133] In at least one embodiment, when selecting an application for use in the deployment pipeline 1510, the user may also select a machine learning model to be used for that particular application. In at least one embodiment, the user may not have a model for use and therefore may select a pre-trained model 1506 for use with the application. In at least one embodiment, the pre-trained model 1506 may not be optimized to produce accurate results with respect to the user's facility's customer dataset 1506 (for example, based on patient diversity, demographics, type of medical imaging device used, etc.). In at least one embodiment, before deploying the pre-trained model 1506 into the deployment pipeline 1510 for use with one or more applications, the pre-trained model 1506 may be updated, retrained, and / or fine-tuned for use at each respective facility.

[0134] In at least one embodiment, the user may select a pre-trained model 1506 to be updated, retrained, and / or fine-tuned, which may be referred to as the initial model 1504 for the training system 1404 within process 1500. In at least one embodiment, a customer dataset 1506 (e.g., imaging data, genomics data, sequencing data, or other data types generated by devices in the facility) may be used to perform model training 1414 (which may include, but is not limited to, transfer learning) with respect to the initial model 1304 to generate an improved model 1512. In at least one embodiment, ground truth data corresponding to the customer dataset 1506 may be generated by the training system 1404. In at least one embodiment, ground truth data may be generated at least partially by clinicians, scientists, physicians, or practitioners in the facility (e.g., as labeled clinic data 1412 in Figure 14).

[0135] In at least one embodiment, AI-assisted annotation 1410 may be used in several examples to generate ground truth data. In at least one embodiment, AI-assisted annotation 1410 (implemented, for example, using the AI-assisted annotation SDK) may leverage a machine learning model (e.g., a neural network) to generate suggestive or predictive ground truth data about a customer dataset. In at least one embodiment, user 1510 may use the annotation tool within a user interface (graphical user interface (GUI)) on computing device 1508.

[0136] In at least one embodiment, user 1510 may interact with a GUI via computing device 1508 to edit or fine-tune (automatic) annotations. In at least one embodiment, polygon editing features may be used to move polygon vertices to more precise or fine-tuned locations.

[0137] In at least one embodiment, if the customer dataset 1506 has relevant ground truth data, the ground truth data (e.g., from AI-assisted annotation, manual labeling, etc.) may be used during model training 1414 to generate an improved model 1512. In at least one embodiment, the customer dataset 1506 may be applied to the initial model 1504 any number of times, and the ground truth data may be used to update the parameters of the initial model 1504 for the improved model 1512 until an acceptable level of accuracy is achieved. In at least one embodiment, once the improved model 1512 is generated, the improved model 1512 may be deployed in one or more deployment pipelines 1510 at the facility to perform one or more processing tasks on medical imaging data.

[0138] In at least one embodiment, the improved model 1512 may be uploaded to a pre-trained model 1506 in a model registry 1424 to be selected by another facility. In at least one embodiment, the process may be completed in any number of facilities, thereby allowing the improved model 1512 to be further improved any number of times with respect to new datasets to generate a more universal model.

[0139] Figure 15B is an illustrative diagram of a client-server architecture 1532 for extending an annotation tool with a pre-trained annotation model, according to at least one embodiment. In at least one embodiment, an AI-assisted annotation tool 1536 may be instantiated based on the client-server architecture 1532. In at least one embodiment, the annotation tool 1536 in an imaging application may assist a radiologist in identifying, for example, organs and anomalies. In at least one embodiment, the imaging application may include, as a non-limiting example, a software tool that helps a user 1510 identify several extreme points on a particular organ of interest in a raw image 1534 (for example, in a 3D MRI or CT scan) and receives automatically annotated results for all 2D slices of the particular organ. In at least one embodiment, the results may be stored in a data store as training data 1538 and used as ground truth data for training (for example, but not limited to). In at least one embodiment, when computing device 1508 sends out extreme points for AI-assisted annotation 1410, a deep learning model, for example, may receive this data as input and return inference results for segmented organs or anomalies. In at least one embodiment, a pre-instantiated annotation tool, such as AI-assisted annotation tool 1536B in Figure 15B, may be extended by making an API call (e.g., API call 1544) to a server, such as annotation support server 1540, which may contain a set of pre-trained models 1542 stored, for example, in an annotation model registry. In at least one embodiment, the annotation model registry may store pre-trained models 1542 (e.g., machine learning models such as deep learning models) that have been pre-trained to perform AI-assisted annotation for specific organs or anomalies. These models may be further updated by using a training pipeline 1504.In at least one embodiment, the pre-installed annotation tool may improve over time as new labeled clinic data 1412 are added.

[0140] Such components can be used to enable custom tasks to be performed using a model, such as a large-scale language model, without the need to retrain the same model.

[0141] Figure 16A shows a data flow diagram of process 1600 for training, retraining, or updating a machine learning model, according to at least one embodiment. In at least one embodiment, process 1600 may be executed using system 1500 of Figure 15 as a non-limiting example. In at least one embodiment, process 1600 may leverage services and / or hardware as described herein. In at least one embodiment, the improved model 1612 produced by process 1600 may be executed by the deployment system for one or more containerized applications in the deployment pipeline.

[0142] In at least one embodiment, model training 1614 may include retraining or updating the initial model 1604 (e.g., a pre-trained model) using new training data (e.g., customer dataset 1606, and / or new input data such as new ground truth data associated with the input data). In at least one embodiment, in order to retrain or update the initial model 1604, one or more output or loss layers of the initial model 1604 may be reset or deleted and / or replaced with one or more updated or new output or loss layers. In at least one embodiment, the initial model 1604 may have parameters (e.g., weights and / or biases) that have been previously fine-tuned and remain from previous training, and therefore training or retraining 1614 may not take as long or require as much processing as training the model from scratch. In at least one embodiment, during model training 1614, by having one or more reset or replaced output or loss layers of the initial model 1604, the parameters may be updated and readjusted for the new dataset based on the loss calculation associated with the accuracy of one or more output or loss layers when generating predictions for the new customer dataset 1606.

[0143] In at least one embodiment, the pre-trained model 1606 may be stored in a data store or registry. In at least one embodiment, the pre-trained model 1606 may be trained at least partially at one or more facilities other than the facility where process 1600 is performed. In at least one embodiment, in order to protect the privacy and rights of patients, subjects, or customers at different facilities, the pre-trained model 1606 may be trained on-site using on-site generated customer or patient data. In at least one embodiment, the pre-trained model 1406 may be trained using the cloud and / or other hardware, but privacy-protected sensitive patient data may not be transferred to any component of the cloud (or other off-site hardware), not be used by those components, or may be inaccessible to those components. In at least one embodiment, if the pre-trained model 1606 is trained using patient data from two or more facilities, the pre-trained model 1606 may be trained individually for each facility and then trained with respect to patient or customer data from another facility. In at least one embodiment, customer or patient data from any number of facilities may be used to train a pre-trained model 1606 on-premises and / or off-premises, such as a data center or other cloud computing infrastructure, if the customer or patient data is released from privacy concerns (e.g., by waiver, for experimental use, etc.) or if the customer or patient data is included in a public data set.

[0144] In at least one embodiment, when selecting an application for use in the deployment pipeline, the user may also select a machine learning model to be used for that particular application. In at least one embodiment, the user may not have a model for use and therefore may select a pre-trained model for use with the application. In at least one embodiment, the pre-trained model may not be optimized to produce accurate results for the user's facility's customer dataset 1606 (based on, for example, patient diversity, demographics, and the type of medical imaging device used). In at least one embodiment, before deploying the pre-trained model into the deployment pipeline for use with one or more applications, the pre-trained model may be updated, retrained, and / or fine-tuned for use at each respective facility.

[0145] In at least one embodiment, the user may select a pre-trained model to be updated, retrained, and / or fine-tuned, which may be referred to as the initial model 1604 for the training system within process 1600. In at least one embodiment, a customer dataset 1606 (e.g., imaging data, genomics data, sequencing data, or other data types generated by devices in the facility) may be used to perform model training (which may include, but is not limited to, transfer learning) on ​​the initial model 1604 to produce an improved model 1612. In at least one embodiment, ground truth data corresponding to the customer dataset 1606 may be generated by the training system 1304. In at least one embodiment, ground truth data may be generated at least partially by clinicians, scientists, physicians, or practitioners in the facility.

[0146] In at least one embodiment, AI-assisted annotation may be used in several examples to generate ground truth data. In at least one embodiment, AI-assisted annotation (for example, implemented using an AI-assisted annotation SDK) may leverage a machine learning model (e.g., a neural network) to generate suggestive or predictive ground truth data about a customer dataset. In at least one embodiment, a user may use an annotation tool within a user interface (graphical user interface (GUI)) on a computing device.

[0147] In at least one embodiment, user 1610 may interact with a GUI via computing device 1608 to edit or fine-tune (automatic) annotations. In at least one embodiment, polygon editing features may be used to move polygon vertices to more precise or fine-tuned locations.

[0148] In at least one embodiment, if the customer dataset 1606 has relevant ground truth data, the ground truth data (e.g., from AI-assisted annotation, manual labeling, etc.) may be used during model training to generate an improved model 1612. In at least one embodiment, the customer dataset 1606 may be applied to the initial model 1604 any number of times, and the ground truth data may be used to update the parameters of the initial model 1604 for the improved model 1612 until an acceptable level of accuracy is achieved. In at least one embodiment, once the improved model 1612 is generated, the improved model 1612 may be deployed in one or more deployment pipelines at a facility to perform one or more processing tasks on medical imaging data.

[0149] In at least one embodiment, the improved model 1612 may be uploaded to a pre-trained model registry, which is to be selected by another facility. In at least one embodiment, the process may be completed in any number of facilities, thereby allowing the improved model 1612 to be further improved any number of times with respect to new datasets to generate a more universal model.

[0150] Figure 16B is an exemplary diagram of a client-server architecture 1632 for extending an annotation tool with a pre-trained annotation model, according to at least one embodiment. In at least one embodiment, an AI-assisted annotation tool 1636 may be instantiated based on the client-server architecture 1632. In at least one embodiment, the annotation tool 1636 in an imaging application may assist a radiologist in identifying, for example, organs and anomalies. In at least one embodiment, the imaging application may include, as a non-limiting example, a software tool that helps a user 1610 identify several extreme points on a particular organ of interest in a raw image 1634 (e.g., in a 3D MRI or CT scan) and receives automatically annotated results for all 2D slices of the particular organ. In at least one embodiment, the results may be stored in a data store as training data 1638 and used as ground truth data for training (e.g., but not limited to). In at least one embodiment, when computing device 1608 sends out extreme points for AI-assisted annotation, a deep learning model, for example, may receive this data as input and return inference results for segmented organs or anomalies. In at least one embodiment, a pre-instantiated annotation tool, such as AI-assisted annotation tool 1636B in Figure 16B, may be extended by making an API call (e.g., API call 1644) to a server, such as annotation support server 1640, which may contain a set of pre-trained models 1642 stored in an annotation model registry, for example. In at least one embodiment, the annotation model registry may store pre-trained models 1642 (e.g., machine learning models such as deep learning models) that have been pre-trained to perform AI-assisted annotation for specific organs or anomalies. These models may be further updated by using a training pipeline.In at least one embodiment, a pre-installed annotation tool may improve over time as new labeled data is added.

[0151] Other variations are within the scope of this disclosure. Thus, the disclosed techniques can be modified and constructed in various ways, some of which are illustrated in the drawings and described in detail above. However, it should be understood that this disclosure is not intended to limit itself to any particular one or more disclosed forms, but rather to encompass all modifications, alternative constructions, and equivalents that fall within the scope and purpose of the disclosure, as defined in the appended claims.

[0152] In the context describing the disclosed embodiments (particularly in the context of the following claims), the terms “a,” “an,” and “the,” and similar demonstrative pronouns, should be interpreted as encompassing both singular and plural, and not as definitions of terms, unless otherwise stated herein or clearly refuted by the context. The terms “comprising,” “having,” “including,” and “containing” should be interpreted as open-ended terms (meaning “including, but not limited to,”) unless otherwise stated. When the term “connected” is unmodified and refers to a physical connection, it should be interpreted as being partially or completely contained, attached, or joined to one another, even if there is something intervening. The detailing of value ranges in this specification is merely intended to serve as a concise way of individually referring to each distinct value that falls within a range, unless otherwise stated herein, and unless each distinct value is incorporated into the specification as if it were individually detailed herein. The use of the terms “set” (e.g., “set of items”) or “subset” should be interpreted as a non-empty set comprising one or more members, unless otherwise stated or denied by the context. Furthermore, unless otherwise stated or denied by the context, the term “subset” of a corresponding set does not necessarily refer to an exact subset of the corresponding set, and a subset and a corresponding set can be equivalent.

[0153] Combinations such as “at least one of A, B, and C” or “at least one of A, B, and C” are understood in contexts generally used to indicate that an item, term, etc., can be either A, B, or C, or any non-empty subset of the set of A, B, and C, unless otherwise specifically stated or explicitly denied by the context. For example, in a descriptive example of a set having three members, the combinations “at least one of A, B, and C” and “at least one of A, B, and C” refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such combinations do not imply as a whole that some embodiments require the presence of each of A, B, and C. Furthermore, unless otherwise stated or negated by the context, the term "plurality" refers to the state of being multiple (for example, "a plurality of items" refers to multiple items). Plurality refers to at least two items, but can be more when explicitly or by contextual indication. Furthermore, unless otherwise stated or clarified by the context, the phrase "based on" means "at least partially based on," and does not mean "based solely on."

[0154] The operation of the processes described herein may be carried out in any preferred order unless otherwise stated herein or explicitly refuted by the context. In at least one embodiment, a process such as the process described herein (or its variations and / or combinations thereof) is carried out under the control of one or more computer systems consisting of executable instructions and is implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or by a combination thereof. In at least one embodiment, the code is stored in a computer-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-temporary computer-readable storage medium that excludes temporary signals (e.g., transient electrical or electromagnetic transmissions that propagate) but includes non-temporary data storage circuit elements (e.g., buffers, caches, and queues) in transceivers for temporary signals. In at least one embodiment, code (e.g., executable code or source code) is stored in a set of one or more non-temporary computer-readable storage media which, when executed by one or more processors of a computer system (i.e., as a result of execution), causes the computer system to perform the operations described herein (or has other memory for storing executable instructions). In at least one embodiment, the set of non-temporary computer-readable storage media comprises a plurality of non-temporary computer-readable storage media which one or more of the individual non-temporary storage media of the plurality of non-temporary computer-readable storage media do not contain all of the code, but the plurality of non-temporary computer-readable storage media collectively contain all of the code.In at least one embodiment, executable instructions are executed such that different instructions are executed by different processors, for example, a non-temporary computer-readable storage medium stores the instructions, the main central processing unit ("CPU") executes some of the instructions, and the graphics processing unit ("GPU") executes others. In at least one embodiment, different components of a computer system have separate processors, and different processors execute different subsets of instructions.

[0155] Accordingly, in at least one embodiment, the computer system is configured to implement one or more services that individually or collectively perform the operations of the processes described herein, and such a computer system consists of applicable hardware and / or software that enables the performance of the operations. Furthermore, a computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment, a distributed computer system comprising multiple devices operating in different ways so that the distributed computer system performs the operations described herein and the single device does not perform all of the operations.

[0156] Any use of any examples or illustrative language provided herein (e.g., "such as") is intended solely to further illustrate the embodiments of this disclosure and, unless otherwise asserted, does not limit the scope of this disclosure. Nothing in this specification should be construed as indicating any unclaimed element as essential to the practice of this disclosure.

[0157] All references cited herein, including publications, patent applications, and patents, are incorporated herein by reference to the same extent as if they were included herein in their entirety, as if each reference were individually and explicitly indicated to be incorporated by reference.

[0158] In the specification and claims, the terms “joined” and “connected” may be used together with their derivatives. It should be understood that these terms may not be intended to be synonymous with one another. Rather, in certain instances, “connected” or “joined” may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with one another. “Joined” may also mean that two or more elements are not in direct contact with one another, but still interact or communicate with one another.

[0159] Unless otherwise specifically stated, throughout this specification, terms such as “processing,” “computing,” “calculating,” or “determining” should be understood to refer to actions and / or processes of a computer or computing system, or similar electronic computing device, that manipulate and / or transform data, represented as electronic or other physical quantities, in the registers and / or memory of the computing system, into other data, similarly represented as physical quantities in the memory, registers, or other such information storage, transmission, or display device of the computing system.

[0160] Similarly, the term “processor” may refer to any device or part of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. In non-limiting examples, “processor” may be a CPU or a GPU. A “computing platform” may comprise one or more processors. As used herein, “software” processes may include software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Each process may also refer to multiple processes for executing instructions serially or in parallel, continuously or intermittently. The terms “system” and “method” are used interchangeably herein only if one or more methods can be embodied by a system, and a method can be considered a system.

[0161] This specification may refer to acquiring, obtaining, receiving, or inputting analog or digital data into subsystems, computer systems, or computer-implemented machines. Acquiring, obtaining, receiving, or inputting analog and digital data can be implemented in various ways, such as by receiving data as parameters to function calls or calls to application programming interfaces. In some implementations, the process of acquiring, obtaining, receiving, or inputting analog or digital data can be implemented by transferring data via serial or parallel interfaces. In other implementations, the process of acquiring, obtaining, receiving, or inputting analog or digital data can be implemented by transferring data via a computer network from a providing entity to a receiving entity. It may also refer to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be implemented by transferring data as input or output parameters to function calls, application programming interfaces, or parameters to inter-process communication mechanisms.

[0162] While the above description outlines exemplary implementations of the techniques described, other architectures may be used to implement the described functionality and are intended to be within the scope of this disclosure. Furthermore, although specific distributions of responsibility are defined above for illustrative purposes, various functions and responsibilities may be distributed and divided in different ways depending on the context.

[0163] Furthermore, while the subject matter is described in language specific to structural features and / or methodological actions, it should be understood that the subject matter claimed in the attached claims is not necessarily limited to the described specific features or actions. Rather, the specific features and actions are disclosed as exemplary forms that implement the claims.

Claims

1. The steps include determining a natural language text string associated with a request based on a request received at an endpoint of a system of at least two or more communicably coupled computing devices, The steps include determining one or more guidance mechanisms associated with the request based on the natural language text string and the task associated with the endpoint, The steps of processing the natural language text string to generate a text result corresponding to the request using a language model and based on at least one or more guidance mechanisms, A step of generating a response to the request using the endpoint, wherein the response includes at least the text result. Methods that include...

2. The method according to claim 1, wherein the endpoint is one of a plurality of endpoints, each of the plurality of endpoints is associated with its own task, and the large language model was not trained to perform at least a subset of the tasks associated with the plurality of endpoints.

3. For each of the plurality of endpoints, the step of selecting each set of one or more guidance mechanisms for each of the tasks of the individual endpoint. The method according to claim 2, further comprising:

4. The method according to claim 1, wherein the one or more guidance mechanisms include a prompt token that indicates at least one of the types of inferences to be performed for the task or the types of results to be returned for the task.

5. The method according to claim 1, wherein the one or more guidance mechanisms include retrieve set tags that indicate one or more datasets to be referenced, and the results are further generated based on using at least the language model to process the data retrieved from the one or more datasets based at least on the retrieve set tags.

6. The method according to claim 1, wherein the one or more guidance mechanisms include adapter weights for modifying at least one of the network weights or layer structure of the language model prior to the processing.

7. The steps include generating one or more alphanumeric strings representing one or more guidance mechanisms, The steps include: prepending the one or more alphanumeric strings to the natural language text string in order to form the modified text string; It further includes, The method according to claim 1, wherein the step of processing the natural language text string includes the step of processing the modified text string.

8. The endpoint is one of a plurality of endpoints, and the method is A step of generating multiple text strings for multiple tasks to be performed using the language model, using one or more of the aforementioned multiple endpoints; The steps include: transmitting the plurality of text strings as at least one of one or more batches or combined homogeneous task streams; The method according to claim 1, further comprising:

9. The method according to claim 1, wherein the language model is associated with two or more model instances of different sizes, and the endpoint is trained with respect to a specified model instance among the two or more model instances to perform the task.

10. The method according to claim 1, wherein the natural language text string is obtained from the request in accordance with one or more marshalling rules used to constitute the endpoint.

11. It is a processor, The aforementioned processor, The system includes determining a natural language text string associated with a request based on a request received at an endpoint of a system of at least two or more communicably coupled computing devices, At a minimum, determining one or more guidance mechanisms associated with the request based on the natural language text string and the task associated with the endpoint, Processing the natural language text string to generate text results corresponding to the request using a large-scale language model and based on at least one or more guidance mechanisms, Using the endpoint, generate a response to the request, wherein the response includes at least the text result. One or more circuits for performing an operation that includes the above A processor equipped with the following features.

12. The processor according to claim 11, wherein the endpoint is one of a plurality of endpoints, each of the plurality of endpoints is associated with its own task, and the large language model is not trained to perform at least a subset of the tasks of the plurality of endpoints.

13. The processor according to claim 12, wherein the operation further comprises, for each of the plurality of endpoints, selecting each set of one or more guidance mechanisms for each of the tasks of the individual endpoint.

14. The aforementioned operation, To generate one or more alphanumeric strings representing one or more of the aforementioned guidance mechanisms, Prepending the one or more alphanumeric strings to the natural language text string in order to form the modified text string It further includes, The processor according to claim 12, wherein the processing of the natural language text string includes processing the modified text string.

15. The processor according to claim 11, wherein the large language model is associated with two or more model instances of different sizes, and the endpoint is trained with respect to a specified model instance among the two or more model instances to perform the task.

16. The aforementioned processor, A system for performing simulation operations. A system for performing simulation operations to test or verify autonomous machine applications. A system for rendering graphical output. A system for performing deep learning operations. Systems implemented using edge devices, A system for generating or presenting virtual reality (VR) content. A system for generating or presenting augmented reality (AR) content. A system for generating or presenting mixed reality (MR) content. A system incorporating one or more virtual machines (VMs), A system that will be implemented, at least partially, in a data center. A system for performing hardware testing using simulation. A system for generating synthetic data. A collaborative content creation platform for 3D assets, or A system implemented using cloud computing resources, at least partially. The processor according to claim 11, comprising at least one of the following.

17. One or more processing units for generating a response to a request, according to the response format associated with the endpoint corresponding to the request. A system comprising the above, wherein the response is generated based on a language model that performs inference according to at least one or more guidance mechanisms determined to be associated with the response format.

18. The system according to claim 17, wherein the one or more guidance mechanisms include at least one of a prompt token, a set of retrieval tags, or an adapter weight.

19. The aforementioned one or more guidance mechanisms Changing one or more weights in at least one layer of the language model, Changing the structure of one or more layers of the aforementioned language model, To update the input to the language model in response to the aforementioned request, or To provide instructions for the data set to be accessed in order to retrieve the data corresponding to the input to the language model. The system according to claim 17, used to carry out at least one of the following.

20. The aforementioned system A system for performing simulation operations. A system for performing simulation operations to test or verify autonomous machine applications. A system for rendering graphical output. A system for performing deep learning operations. Systems implemented using edge devices, A system for generating or presenting virtual reality (VR) content. A system for generating or presenting augmented reality (AR) content. A system for generating or presenting mixed reality (MR) content. A system incorporating one or more virtual machines (VMs), A system that will be implemented, at least partially, in a data center. A system for performing hardware testing using simulation. A system for generating synthetic data. A collaborative content creation platform for 3D assets, or A system implemented using cloud computing resources, at least partially. The system according to claim 17, comprising at least one of the following.

Images