Automatic model card generation for machine learning models

Abstract

The invention relates to automatic model card generation for machine learning models. In examples, automatic generation of model cards for machine learning models is described herein. Systems and methods are disclosed that automatically generate a model card to associate with a model (e.g., a machine learning model, an AI model, a neural network, etc.) using one or more language models that process input data representing information associated with the model. As described herein, information associated with the model may include at least a portion of source code for generating the model, one or more documents describing the model, one or more previously generated model cards, and / or any other information associated with the model. Further, in some examples, additional data may be input into the language model to generate a model card, such as data representing questions used to retrieve related information and / or data representing reference information associated with one or more other models.

Description

BACKGROUND

[0001] Models (e.g., machine learning models, neural networks, etc.) can be used in a variety of applications, including but not limited to healthcare, finance, transportation, manufacturing, and / or entertainment. For example, in a healthcare-related context, AI-powered systems can help diagnose diseases, analyze medical images, and / or personalize treatment plans. In contrast, models for a transportation-related context can enable machines (e.g., semi-autonomous and / or fully autonomous vehicles) to perceive their surroundings and navigate safely. Thus, different models can be suitable for different uses and / or have different strengths and weaknesses, even when comparing different models within the same context (e.g., transportation).

[0002] To help understand the capabilities, limitations, and / or differences between models, end users can evaluate model cards associated with the models. For example, a model card can contain various information about a particular model, such as the development process of the model, training data, performance metrics, potential biases, limitations, intended use cases, and / or out-of-scope applications, which can allow end users to make informed decisions about the deployment and / or use of the model. Additionally, such model cards can also help support compliance with regulatory standards and / or industry best practices. As such, organizations can use model cards to demonstrate compliance with various requirements, such as legal requirements, corporate compliance requirements, and / or ethical requirements, which can help ensure that AI systems are developed and / or deployed in a manner that is consistent with societal values and norms.

[0003] Accordingly, conventional systems can use various tools and / or platforms to attempt to generate model cards for models, such as the Model Cards Toolkit, the Python Toolkit, Papers with Code, the HuggingFace Model Card Generator, and others. However, for each of these tools and / or platforms, users need to manually search for information describing the model in order to input relevant portions of the information into the model card. Additionally, when these models are updated, such as by further training, these conventional systems require users to update the model card in order to keep the information accurate. Thus, generating and / or updating model cards can require significant human resources and / or time. Additionally, generating and / or updating model cards can be prone to errors, such as errors resulting from users inputting inaccurate information and / or errors resulting from model cards not being updated to reflect current versions of the models. SUMMARY

[0004] Embodiments of the present disclosure relate to automatic model card generation for machine learning models. Systems and methods are disclosed for automatically generating model cards using one or more language models that process input data representing information associated with a model (e.g., a machine learning model, an AI model, a neural network, etc.). As described herein, the information associated with the model can include at least a portion of the source code used to generate the model, one or more documents describing the model, one or more previously generated model cards, and / or any other information associated with the model. Additionally, in some examples, additional data can be input into the language model(s) to generate the model card, such as data representing a query (e.g., a question) to retrieve relevant information needed to generate the model card, data representing the format of the model card (e.g., if this is a new model card), and / or data representing reference information associated with one or more other models.

[0005] In comparison to conventional systems such as the conventional system described above, the systems of the present disclosure can automatically generate model cards for models using language model(s). Thus, in comparison to conventional systems, a user can not need to manually identify information needed to generate a model card and / or input relevant portions of the information into the model card, which can save time and / or computational resources. Additionally, in comparison to conventional systems, the systems of the present disclosure can be used to automatically update model cards, such as when a model is updated (e.g., the model is further trained to make it more accurate and / or perform additional processing tasks), without the user needing to input the information into the model card again. By automatically generating and / or updating model cards in this way, the model cards can also be more up-to-date and more accurate, as the generation and / or updating of the model cards is less prone to user error, as described herein. BRIEF DESCRIPTION OF DRAWINGS

[0006] The present systems and methods for automatic model card generation for machine learning models are described in detail below with reference to the accompanying drawings, wherein:

[0007] FIG. 1A An example of a process for generating a new model card associated with a model is shown in accordance with some embodiments of the present disclosure;

[0008] FIG. 1B An example of a process for updating a model card associated with a model is shown in accordance with some embodiments of the present disclosure;

[0009] FIG. 1C An example of a process for verifying a model card associated with a model is shown in accordance with some embodiments of the present disclosure;

[0010] FIG. 2An example of retrieving information for generating a model card associated with a model is shown in accordance with some embodiments of the present disclosure;

[0011] FIG. 3 An example of retrieving reference information for generating a model card associated with a model is shown in accordance with some embodiments of the present disclosure;

[0012] FIG. 4 An example of a model card that can be associated with a model is shown in accordance with some embodiments of the present disclosure;

[0013] FIG. 5 An example of updating a model card associated with a model from an example is shown in accordance with some embodiments of the present disclosure; FIG. 4

[0014] FIG. 6 An example of one or more systems that can perform various tasks using model cards is shown in accordance with some embodiments of the present disclosure;

[0015] FIG. 7 A flowchart showing a method for generating a new model card associated with a model is shown in accordance with some embodiments of the present disclosure;

[0016] FIG. 8 A flowchart showing a method for updating a model card associated with a model is shown in accordance with some embodiments of the present disclosure;

[0017] FIG. 9 A flowchart showing a method for verifying a model card associated with a model is shown in accordance with some embodiments of the present disclosure;

[0018] FIG. 10 A flowchart showing a method for generating a model card that is then used to determine whether to provide a model to one or more computing devices is shown in accordance with some embodiments of the present disclosure;

[0019] FIG. 11A is a block diagram of an example generative language model system suitable for implementing some embodiments of the present disclosure;

[0020] FIG. 11B is a block diagram of an example generative language model including a transformer encoder-decoder suitable for implementing some embodiments of the present disclosure;

[0021] FIG. 11C is a block diagram of an example generative language model including a decoder-only transformer architecture suitable for implementing some embodiments of the present disclosure;

[0022] FIG. 12 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and​

[0023] FIG. 13 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0024] Systems and methods related to automatic model card generation for machine learning models are disclosed. For example, the system(s) can obtain, receive, retrieve, and / or store data associated with one or more models within one or more databases. As described herein, the data for a model can include, but is not limited to, data for executing the model, data representing one or more model cards associated with the model (e.g., if previously generated), and / or data representing information associated with the model, such as source code associated with the model, one or more documents describing the model (e.g., one or more research papers, one or more abstracts, etc.), and / or any other information. When describing information associated with a model, to distinguish between different types of information that can be used to generate a model card, the information associated with the model for which a model card is being generated can be referred to as “primary information,” while the information associated with one or more other reference models can be referred to as “reference information.”

[0025] In some examples, the system(s) can store at least a portion of the data using one or more formats. For example, the system(s) can segment information into individual portions, such as letters, numbers, words, sentences, paragraphs, code snippets, and / or any other portions of text, where the portions of information can be referred to as “chunks” of information. The system(s) can then process the chunks of information using one or more embedding components (e.g., one or more machine learning models, one or more neural networks, one or more transformers, one or more encoders, etc.) in order to generate an embedding (and / or vector) associated with the chunk of information. Additionally, the system(s) can then store the embedding and / or the chunk of information in the database(s). As described further herein, the system(s) can store information using such formats in order to improve later processing for extracting relevant portions of information.

[0026] In some examples, the system(s) can then use the data associated with the one or more models to generate at least a model card associated with the model. For example, the system(s) can obtain, receive, retrieve, and / or store data representing queries (e.g., questions) associated with extracting primary information associated with the model. In some examples, at least a portion of the queries can be specific to a format associated with the model card being generated, such as one or more queries to extract primary information related to fields included in the model card. For example, the queries can be to extract primary information related to attributes associated with the model, intended use cases for the model, out-of-scope applications for the model, inputs for the model, outputs for the model, intended users for the model, how the model will perform with different groups, training of the model, limitations of the model, computational requirements of the model, and / or the like. Additionally or alternatively, in some examples, at least a portion of the queries can be specific to a type of the model, such as one or more queries to extract primary information related to use (e.g., task) of the model (e.g., transportation-related model queries, language-related model queries, and / or the like).

[0027] The system(s) can then use the queries to retrieve at least a portion of the primary information associated with the model. For example, the system(s) can generate an embedding associated with a query using the embedding component(s). The system(s) can then use the generated embedding to identify stored embeddings associated with primary information related to the query. For example, for a generated embedding associated with a query, the system(s) can use the generated embedding to identify a number of stored embeddings that are most closely related to the generated embedding. As described herein, the number of stored embeddings can include, but is not limited to, one embedding, two embeddings, five embeddings, ten embeddings, twenty embeddings, and / or any other number of embeddings. In some examples, the embeddings and / or the primary information (e.g., source code, documentation, model cards, and / or the like) associated with the embeddings can be referred to as “primary blocks.”

[0028] The system(s) can then perform a first processing task based at least in part on the first call, the first processing task including processing input data using one or more language models. As described herein, the input data can represent at least the primary information associated with the model (e.g., source code, documentation, and / or the like), the identified primary information blocks, and / or the query. In some examples, the input data can represent actual text associated with the primary information, the identified primary information blocks, and / or the query. Additionally or alternatively, in some examples, the input data can represent embeddings associated with the primary information, the identified primary information blocks, and / or the query. Further, in some examples, the input data can also represent a prompt, such as a prompt to generate and / or output particular data. In any of these examples, the language model(s) can generate an initial output, such as an initial output representing information associated with the query.

[0029] In some examples, the system(s) can then use the primary information (e.g., the primary chunk) associated with the model to retrieve reference information associated with one or more reference models. As described herein, the system(s) can use one or more techniques to retrieve the reference information. For example, the system(s) can use a retrieved embedding associated with the primary information to identify a number of additional embeddings associated with reference information that is most closely related to the retrieved embedding. As described herein, the number of additional embeddings can include, but is not limited to, one embedding, two embeddings, five embeddings, ten embeddings, twenty embeddings, and / or any other number of embeddings. In some examples, a portion of the additional embeddings and / or reference information (e.g., source code, documentation, model cards, etc.) associated with the additional embeddings can be referred to as a “reference chunk.”

[0030] In some examples, for example, if the system(s) are generating a new model card for a model, the system(s) can retrieve a model card template that represents the format of a model card. For example, the model card template can indicate fields for different types of information included in a model card, such as fields for attributes, intended use cases, out-of-scope applications, inputs, outputs, intended users, model performance for different groups, training, limitations, computational requirements, etc. As described herein, attributes can include, but are not limited to, a name and / or identifier for the model, one or more names and / or identifiers for one or more datasets used to train the model, one or more sizes for the dataset(s), a number of epochs used for training, a type of license associated with the model, one or more risk scores associated with the model, one or more bias scores associated with the model, one or more losses associated with the model, and / or any other type of attribute. Additionally or alternatively, in some examples, for example, when the system(s) are updating a previously generated model card for a model, the system(s) can retrieve an existing model card from the database(s).

[0031] The system(s) can then perform a second processing task based on the second call, for example, including processing additional input data using the language model(s), as described herein. The additional input data can represent, at least, the initial output from the language model(s) during the first processing task (e.g., information associated with the query), reference information associated with the reference model(s) (e.g., portions of the source code, documentation, model cards, etc.), previously generated model cards, and / or model card templates. In some examples, the additional input data can represent actual text associated with the initial output, the reference information, the previously generated model cards, and / or the model card templates. Additionally or alternatively, in some examples, the additional input can represent embeddings associated with the initial output, the reference information, the existing model cards, and / or the model card templates.

[0032] In examples in which the system(s) are generating a new model card for a model, the language model(s) can generate output representing the model card. For example, the model card can include a format associated with the model card template, for example, by including information associated with each field. However, in examples in which the system(s) are updating an existing model card for a model, the language model(s) can generate output representing that the existing model card is updated. For example, the updated model card can include updated information for one or more of the fields. For example, if the model is further trained using a new dataset, the model card can be updated to include information associated with the further training and / or the new dataset. The system(s) can then store the new model card and / or the updated model card in association with the model, for example, in a database.

[0033] While these examples describe using the language model(s) to generate a new model card and / or update an existing model card, in other examples, the system(s) can use the language model(s) to perform one or more additional processes with respect to a model card, such as verifying an existing model card. For example, during the second processing task, the language model(s) can process additional input data representing the initial output from the language model(s), the reference information, and / or the existing model card. Based at least on the processing, the language model(s) can determine whether the information included in the existing model card is accurate. Additionally, the language model(s) can then generate output indicating (1) that the existing model card is not verified if the information is not accurate, or (2) that the existing model card is verified if the information is accurate. In such examples, if the existing model card is not verified, the language model(s) can further output data indicating which information associated with the existing model card is inaccurate and / or updated information representing the model card.

[0034] Additionally, while examples herein describe generating a single model card associated with a single model, in other examples, a similar process can be used to generate any number of model cards associated with any number of models. For a first example, when multiple models are included in a processing pipeline, the system(s) can perform a similar process to generate a single model card associated with the processing pipeline. In such examples, the model card can include information describing the individual models included in the pipeline and / or information describing the entire pipeline. For a second example, and again when multiple models are included in a processing pipeline, the system(s) can generate multiple model cards associated with the pipeline. In such examples, one or more of the model cards can include information describing one or more of the models, and / or the model cards can include information describing the entire pipeline.

[0035] In some examples, the system(s) can use a model card for a model (e.g., a new model card, an updated model card, etc.) to perform one or more additional processes. For example, the system(s) can receive a request from one or more endpoints to execute a model on one or more devices associated with the endpoint(s). In some examples, the request can indicate a particular model of the model(s) that the endpoint(s) are requesting to execute. Based at least on the request, the system(s) can obtain at least a model card stored in association with the particular model. Using the model card and known information about the requesting endpoint(s), the system(s) can determine whether to provide the model to the endpoint(s).

[0036] In some examples, the system(s) can evaluate the attribute(s) and / or other information included in a model card relative to one or more criteria associated with the endpoint(s). For example, the criteria can include a policy (e.g., an enterprise policy, etc.) associated with the endpoint(s) that indicates various requirements for the model(s) that can be used. For example, the policy can indicate, among other things, a risk threshold for a model, a permission requirement for a model, a training requirement for a model, etc. Additionally or alternatively, the criteria can include a hardware specification that indicates one or more limitations and / or capabilities associated with the device(s) of the endpoint(s) that are to execute the model(s). For example, the hardware specification can indicate features associated with one or more processors of the device(s) (e.g., a processor type, a processor brand, a processor model, etc.), memory limitations and / or capabilities associated with the device(s), version numbers associated with the device(s), etc.

[0037] In some examples, the system(s) can determine that the endpoint(s) and / or the device(s) are permitted and / or capable of performing the requested model. For example, based at least on the evaluation, the system(s) can determine that the model complies with a given set of requirements (e.g., which can be indicated in a policy), the model is optimized for the endpoint’s execution environment, and the device hardware is capable of properly executing the model. The system(s) can then send data to the endpoint for executing the model on the device(s). Additionally or alternatively, if the system(s) determine that the endpoint(s) and / or the device(s) are prohibited from executing the model, the system(s) can send an indication to the endpoint(s). In some examples, the indication can indicate one or more reasons why the model is prohibited from being executed on the endpoint(s). For example, the indication can indicate that the policy restricts the endpoint(s) from executing the requested model, and / or the capabilities / limitations of the device(s) can prohibit the requested model from being executed.

[0038] For example, the model card can indicate a risk score associated with the requested model, and the system(s) can evaluate this risk score against a risk threshold associated with the endpoint(s) (e.g., indicated in a policy). Based at least on the evaluation, the system(s) can determine whether to provide the endpoint(s) with data for executing the model. That is, if the model risk score meets or exceeds the risk threshold, the system(s) can determine to prohibit the model from being executed on the endpoint(s). However, if the model risk score is less than the risk threshold, the system(s) can determine to allow the endpoint(s) to execute the model.

[0039] As another example, the system(s) can determine one or more thresholds corresponding to one or more hardware capabilities for executing the model based at least on the model card. Example thresholds can include, but are not limited to, a central processing unit (CPU) threshold, a graphics processing unit (GPU) threshold, a data processing unit (DPU) threshold, a network hardware unit threshold, a memory threshold, and a network bandwidth threshold. The system(s) can then evaluate actual capabilities associated with the device(s) of the endpoint(s) against the one or more hardware thresholds to determine whether to provide the endpoint(s) with data for executing the model. If the system(s) determine that the actual capabilities meet or exceed the threshold(s), the system(s) can determine to provide the model to the endpoint(s). However, if the actual capabilities do not meet the threshold(s), the system(s) can determine to prohibit the model from being executed by the endpoint(s).

[0040] In some examples, the system(s) can offer one or more alternative (e.g., more suitable, more capable, and the like) models to the endpoint(s). In some examples, the alternative model(s) can be offered to the endpoint(s) based at least on determining that the endpoint(s) are blocked from executing the requested model. Additionally or alternatively, the endpoint(s) can query the system(s) to find a model(s) that satisfy certain preconditions, for an intended purpose, and the like. As an example and not by way of limitation, the endpoint(s) can request a model for detecting objects in a machine’s environment, which has been trained using a closed (e.g., non-open source) dataset, and optimized for a rural environment. Based on this request, the system(s) can evaluate one or more model cards for one or more offered models that will satisfy these requirements. In some examples, the system(s) can further provide the model card(s) corresponding to the offered model(s) to the endpoint(s), and the endpoint(s) can select which model(s) to execute.

[0041] The systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, dirigibles, watercraft, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater boats, drones, and / or other vehicle types. Further, the systems and methods described herein can be used for various purposes, such as, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twin, autonomous or semi-autonomous machine applications, deep learning, environment simulation, object or actor simulation and / or digital twin, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable application.

[0042] The disclosed embodiments can be included in various different systems, such as, for example, automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, shipping systems, smart regional monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing large language models (LLMs), systems implementing one or more visual language models (VLMs), systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in data centers, systems for performing conversational AI operations, systems for performing optical transport simulation, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0043] Reference is made to FIG. 1A , FIG. 1A An example of a process 100 for generating a new model card associated with a model according to some embodiments of the present disclosure is shown. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be omitted altogether. Further, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For instance, various functions can be performed by a processor executing instructions stored in memory.

[0044] The process 100 can include the one or more generation components 102 sending model identifier data 104 associated with the model 106 to one or more language model (LM) components 108. As described herein, in some examples, the model identifier data 104 can represent any type of identifier associated with the model 106, such as a name, a label, an alpha identifier, a numeric identifier, an alphanumeric identifier, a code, and / or the like. Additionally, in some examples, the model identifier data 104 can represent information for locating and / or retrieving the model 106 and / or primary information associated with the model 106. For example, the model identifier data 104 can represent a uniform resource locator (URL) and / or any other type of locator for retrieving the model 106 and / or the primary information associated with the model 106. Based at least on receiving the model identifier data 104, the LM component(s) 108 can then send the model identifier data 104 to one or more model loader components 110.

[0045] The process 100 can then include the model loader component(s) 110 using the model identifier data 104 to retrieve primary information associated with the model 106. For example, and as illustrated, one or more model databases 112 can store primary information associated with the model 106, such as source code 114 associated with the model 106 and / or one or more documents 116 associated with the model 106. As described herein, the documents 116 can include, but are not limited to, research papers, articles, abstracts, manuals, texts, and / or any other source of information associated with the model 106. Additionally, at least a portion of the primary information can describe attributes, intended use cases, out-of-scope applications, inputs to the model, outputs of the model, intended users, model performance for different groups, training, limitations, computational requirements, and / or any other information associated with the model 106. Accordingly, the model loader component(s) 110 can use the model identifier data 104 to retrieve at least the source code 114 and the document(s) 116 from the model database(s) 112.

[0046] In some examples, the process 100 can include one or more extraction components 118 segmenting the source code 114 and / or the document(s) 116 into chunks, such as letters, numbers, words, sentences, paragraphs, code snippets, and / or any other portion of text. The extraction component(s) 118 can then store the chunks in one or more databases 120 (e.g., the extraction component(s) 118 can perform code ingestion). In some examples, the extraction component(s) 118 can perform additional and / or alternative processes, such as generating embeddings associated with the chunks using one or more embedding components 122. In such examples, the extraction component(s) 118 can further store the embeddings in the database(s) 120.

[0047] Process 100 may then include sending a request from (multiple) model loader components 110 to (multiple) extraction components 118 to extract at least a portion of the retrieved information, wherein the request may be represented by request data 124. As described herein, request data 124 may represent one or more queries related to a specific type of information to be extracted. In some examples, at least a portion of the query may be specific to the format associated with the model card being generated, such as extracting one or more questions about the main information that should be included within the model card. For example, the query may be for extracting information about attributes associated with model 106, intended use cases for model 106, out-of-scope applications of model 106, intended users of model 106, how model 106 will be performed in different groups, training of model 106, limitations of model 106, computational requirements of model 106, etc. Additionally, in some examples, at least a portion of the query may be specific to the type of model 106, such as extracting one or more queries related to the use of model 106 (e.g., transportation-related model, language-related model, etc.).

[0048] For the first example, for instance, to retrieve key information associated with the functionality of source code 114, the query could include “Can you describe the functionality of the code?” For the second example, for instance, to retrieve key information associated with the description of model 106, the query could include “Describe what this model does, including supporting available images and articles.” For the third example, for instance, to retrieve key information associated with the architecture of model 106, the query could include “What type of neural network architecture is used in the model?” For the fourth example, for instance, to retrieve key information associated with the input type of model 106, the query could include “What type of input data does the model expect—audio, images, text, or any other data?” Still, for the fifth example, for instance, to retrieve key information associated with the training set of model 106, the query could include “Which datasets were used to train the model?” While these are just a few exemplary queries for retrieving key information associated with model 106, in other examples, additional and / or alternative queries can be used to retrieve key information of additional and / or alternative types associated with model 106.

[0049] To extract portions of the primary information (also referred to as chunks), the extraction component(s) 118 can use the embedding component(s) 122 to generate embeddings associated with individual chunks of the primary information. As described herein, the embedding component(s) 122 can include and / or use one or more machine learning models, one or more neural networks, one or more transformers, one or more encoders, and / or any other type of component configured to partition the primary information into chunks and / or generate embeddings associated with the chunks. The extraction component(s) 118 can then store the embeddings (and / or chunks) in the database(s) 120, e.g., a vector database (and / or any other type of database).

[0050] Additionally, the extraction component(s) 118 can use the embedding component(s) 122 to generate an embedding associated with a query represented by the request data 124 (also referred to as a “query embedding”). The extraction component(s) 118 can then use the query embedding to retrieve one or more chunks of information related to generating a model card associated with the model 106. For example, and with respect to the query embedding, the extraction component(s) 118 can analyze the query embedding with respect to the embeddings stored in the database(s) 120 to identify a number of embeddings related to the query embedding. As described herein, the number of embeddings can include, but is not limited to, one embedding, two embeddings, five embeddings, ten embeddings, twenty embeddings, and / or any other number of embeddings. Additionally, the extraction component(s) 118 can perform any technique to identify the number of embeddings, e.g., identify the embeddings most closely related to the query embedding based at least on a distance between vectors associated with the embeddings and a vector associated with the query embedding in a latent space.

[0051] For example, FIG. 2 An example of retrieving information for generating a model card associated with a model is shown in accordance with some embodiments of the present disclosure. As shown, the extraction component(s) 118 can generate embeddings 202(1)-(N) (also referred to as “embedding 202” in the singular and “embeddings 202” in the plural) associated with chunks 204(1)-(N) of primary information (also referred to as “chunk 204” in the singular and “chunks 204” in the plural). As described herein, the chunks 204 can include a portion of the primary information, e.g., a portion of source code and / or a portion of a document associated with the model. Additionally, the extraction component(s) 118 can generate embeddings 206(1)-(O) (also referred to as “embedding 206” in the singular and “embeddings 206” in the plural) associated with queries 208(1)-(O) (also referred to as “query 208” in the singular and “queries 208” in the plural).

[0052] The extraction component(s) 118 can then analyze the embeddings 206 with respect to the embeddings 202 in order to identify at least a portion of the embeddings 202 that are relevant to the embeddings 206. For example, and as shown, the extraction component(s) 118 can perform one or more techniques to determine that the embeddings 202(2)-(4) are relevant to the embedding 206(1). For example, the extraction component(s) 118 can determine that the embeddings 202(2)-(4) include the embeddings 202 that are closest to the embedding 206(1) within the latent space. In other words, the extraction component(s) 118 can use the embeddings 202(2)-(4) and the embedding 206(1) to determine that the primary information blocks 204(2)-(4) are relevant to the query 208(1). The extraction component(s) 118 can then perform a similar process for each of the other embeddings 206(2)-(O).

[0053] Referring back to the example of FIG. 1, FIG. 1A In some examples, the block data 126 can represent the actual blocks of primary information (e.g., one or more portions of the source code 114, one or more portions of the document(s) 116, etc.), while in some examples, the block data 126 can represent embeddings associated with the blocks. The process 100 can then include the model loader component(s) 110 providing the block data 126 to the LM component(s) 108. Additionally, in some examples, the process 100 can include the LM component(s) 108 receiving prompt data 128, where the prompt data 128 represents at least a prompt to extract information used to generate the model card. For example, the prompt can indicate one or more language models 130 to identify necessary information to generate the model card, one or more indications of one or more fields of the model card for which information needs to be retrieved, one or more indications of data (e.g., the block data 126) for which information can be retrieved, and / or any other instructions associated with retrieving the information.

[0054] The process 100 can then include the language model(s) 130 processing at least the prompt data 128 and the block data 126 during a first processing task, where the first processing task can be associated with the first call 132(1). In some examples, the language model(s) 130 can perform any type of processing, such as described herein with respect to the language model(s) 130 of FIG. 1. FIG. 11A-11CThe processing. Based at least on the processing, process 100 can include language model(s) 130 generating and / or outputting data 134 associated with the first processing task. For example, output data 134 can represent at least a portion of the primary information that generating a model card associated with model 106 can require. For example, output data 134 can represent primary information related to attributes associated with model 106, intended use cases for model 106, out-of-scope applications for model 106, inputs for model 106, outputs for model 106, intended users for model 106, how model 106 will perform with different groups, training of model 106, limitations of model 106, computational requirements of model 106, and / or any other information that can be included in a model card.

[0055] In some examples, because FIG. 1A In examples associated with generating a new model card associated with model 106, process 100 can include model loader component(s) 110 receiving template data 136 representing a model card template for generating the new model card. As described herein, a model card template can represent a format of a model card, such as fields of information to be included in the model card and / or a layout of the fields (e.g., an order in which the fields are included within the model card). For example, a model card template can indicate whether information related to attributes associated with model 106, intended use cases for model 106, out-of-scope applications for model 106, inputs for model 106, outputs for model 106, intended users for model 106, how model 106 will perform with different groups, training of model 106, limitations of model 106, computational requirements of model 106, and / or any other information that can be included in a model card is to be included.

[0056] Process 100 can then include model loader component(s) 110 sending template data 136 to generation component(s) 102, which in turn send template data 136 to LM component(s) 108. In this way, LM component(s) 108 can use template data 136 to determine a format for generating a model card associated with model 106, which is described in more detail herein.

[0057] In some examples, and as FIG. 1AAs further shown by the example of FIG. 1, the process 100 can include the LM component(s) 108 extracting additional information, such as reference information, using one or more reference extraction components 138. For example, the reference extraction component(s) 138 can store reference information 140 associated with one or more reference models. As described herein, the reference information 140 associated with a reference model can include, but is not limited to, source code associated with the reference model, one or more documents associated with the reference model, a model card associated with the reference model, and / or any other information. Additionally, similar to the documents 116, the documents associated with the reference information 140 can include, but are not limited to, research papers, articles, abstracts, manuals, texts, and / or any other source of information associated with the reference model.

[0058] In some examples, the reference extraction component(s) 138 can then generate embeddings associated with portions of the reference information 140 using one or more embedding components 142, where the portions of the reference information 140 can be referred to as“reference chunks” of the reference information 140. As described herein, the embedding component(s) 142 can include and / or use one or more machine learning models, one or more neural networks, one or more transformers, one or more encoders, and / or any other type of component configured to partition the reference information 140 into reference chunks and / or generate embeddings associated with the reference chunks. The reference extraction component(s) 138 can then store the embeddings (and / or reference chunks) in one or more databases 144, such as a vector database (and / or any other type of database).

[0059] The reference extraction component(s) 138 can then use the chunk data 126 to retrieve one or more reference chunks related to generating a model card associated with the model 106. For example, and for each chunk and / or embedding represented by the chunk data 126, the reference extraction component(s) 138 can analyze the respective embedding relative to the embeddings stored in the database(s) 144 to identify a number of embeddings related to the respective embedding. As described herein, the number of embeddings can include, but is not limited to, one embedding, two embeddings, five embeddings, ten embeddings, twenty embeddings, and / or any other number of embeddings. Additionally, the reference extraction component(s) 138 can perform any technique to identify the number of embeddings, such as identifying the embeddings most closely related to the respective embedding based on a distance between a vector associated with the embedding and vectors associated with the respective embedding in a latent space.

[0060] For example, FIG. 3Examples of retrieving reference information for generating model cards associated with a model, according to some embodiments of the present disclosure, are shown. As shown, (a plurality of) reference extraction components 138 can generate embeddings 302(1)-(Q) (also referred to as "embedded 302" in the singular or "a plurality of blocks 304" in the plural) associated with reference blocks 304(1)-(Q) of the reference information. As described herein, reference blocks 304 may include a portion of reference information 140, such as a portion of source code, a portion of documentation, and / or a portion of a model card associated with a reference model. Additionally, the (multiple) reference extraction unit 138 may receive block data 306 (which may represent and / or resemble block data 126) representing an embedding 308 (1)-(R) (also referred to as "embedding 308" in the singular or "multiple embeddings 308" in the plural) associated with the main information block 310 (1)-(R) (also referred to as "block 310" in the singular or "multiple embeddings 308" in the plural).

[0061] The (multiple) reference extraction components 138 can then analyze the embedding 308 relative to the embedding 302 to identify at least a portion of the embedding 302 that is related to the embedding 308. For example, and as shown, the (multiple) reference extraction components 138 can perform one or more techniques to determine that the embeddings 302(2)-(4) are related to the embedding 308(1). For example, the (multiple) reference extraction components 138 can determine that the embeddings 302(2)-(4) include the embedding 302 that is closest to the embedding 308(1) in the latent space. In other words, the (multiple) reference extraction components 138 can use the embeddings 302(2)-(4) and the embedding 308(1) to determine that the reference blocks 304(2)-(4) of the reference information can be related to the main block 310(1) of the main information associated with the model for which the model card is being generated. The (multiple) reference extraction components 138 can then perform a similar process for each of the other embeddings 308(2)-(S).

[0062] Return to reference FIG. 1A For example, process 100 may include multiple LM components 108 retrieving block data 146 from multiple reference extraction components 138. In some examples, block data 146 may represent reference blocks of reference information 140, such as one or more portions of source code, one or more portions of document(s), one or more portions of model(s), and / or any other portion of reference information 140. Alternatively or additionally, in some examples, block data 146 may represent an embedding associated with a reference block of reference information 140.

[0063] Process 100 can then include language model(s) 130 processing at least output data 134, template data 136, and / or chunk data 146, for example, during a second processing task associated with second call 132(2). In some examples, language model(s) 130 can perform any type of processing, such as described herein with respect to FIG. 11A-11C the processing described. Based at least on the processing, process 100 can include language model(s) 130 generating and / or outputting card data 148 (e.g., metadata, etc.) associated with the second processing task. For example, card data 148 can represent at least a model card associated with model 106. As described herein, since the model card is generated using at least output data 134 and template data 136, the model card can include the format of the template model card and information represented by output data 134.

[0064] For example, FIG. 4 An example of a model card 402 (which can be represented by card data 148) that can be associated with a model in accordance with some embodiments of the present disclosure is shown. As shown, model card 402 can include individual fields 404(l)-(S) (also referred to as "field 404" in the singular or "fields 404" in the plural), where each field 404 can be associated with a type of information corresponding to the model. For example, fields 404 can be associated with: attributes associated with the model (e.g., a name and / or identifier of the model, a name and / or identifier of a dataset, a size of the dataset, etc.), an intended use case of the model, an out-of-scope application of the model, an input of the model, an output of the model, an intended user of the model, how the model will be executed in different groups, a training of the model, a limitation of the model, a computational requirement of the model, and / or any other information that can be included in model card 402. Model card 402 can then include information 406(l)-(S) (also referred to as "information 406") describing the information of the fields.

[0065] For example, field 404 may be associated with a description of the model, and information 406 may describe the model (e.g., using text, images, videos, etc.); field 404 may be associated with a license and / or terms of use of the model, and information 406 may describe the license and / or terms; field 404 may be associated with a model architecture, and information 406 may describe the model architecture (e.g., describing the neural network associated with the model, such as its type); field 404 may be associated with the model's inputs, and information 406 may describe the type of the model's inputs (e.g., text, images, tokens, audio, etc.); field 404 may be associated with the model's outputs, and information 406 may describe the type of the outputs and / or details about the outputs; field 404 may be associated with a version of the model, and information 406 may describe the version; and / or field 404 may be associated with a dataset used to train the model, and information 406 may describe the dataset (e.g., the dataset's identifier, how the data examples in the dataset were collected, how the dataset was labeled, how the dataset was tested, how the dataset was evaluated, etc.). While these are just a few examples of the fields that can be included in Model 402, in other examples, Model 402 can include any other type of field.

[0066] As described in this article, in some examples, model cards may have already been generated for model 106, where the model cards need to be updated based on the occurrence of one or more events. For example, if model 106 is updated, for instance, with a new name, new training (e.g., a new dataset), new intended use, new constraints, new computational requirements, etc., then the model cards may need to be updated to reflect one or more of these updates to model 106. Therefore, FIG. 1B An example of a process 150 for updating a model card associated with a model, according to some embodiments of the present disclosure, is shown.

[0067] As shown in the figure, process 150 can be similar to process 100, except that... FIG. 1B In the example, model 106 may have been associated with a previously generated model card, which is represented by card data 152. Therefore, it is not like... FIG. 1A Instead of receiving template data 136 as in the example, the model loader component 110 may alternatively retrieve card data 152 representing a previously generated model card and send the card data 152 to the generation component 102. Furthermore, the generation component 102 may then send the card data 152 to the LM component 108, which may then use the card data 152 to, for example, update the previously generated model card with new information.

[0068] For example, process 150 may include (multiple) language models 130, for example, processing at least output data 134, block data 146, and card data 152 during a second processing task associated with the second call 132(2). In some examples, (multiple) language models 130 may perform any type of processing, such as those described herein. FIG. 11A-11C The aforementioned processing. Based at least on the aforementioned processing, process 150 may include language models(s)130 generating and / or outputting updated card data 154 associated with the second processing task. For example, updated card data 154 may at least indicate that a previously generated model card associated with model 106 has been updated. As described herein, in some examples, language models(s)130 may update information associated with one or more fields of the model card. For example, language models(s)130 may update information associated with (e.g., the name and / or identifier of the model, the name and / or identifier of the dataset, the size of the dataset, etc.) attributes associated with model 106, (e.g., the name and / or identifier of the model, the name and / or identifier of the dataset, the size of the dataset, etc.), (e.g., the intended use cases of model 106, the (e.g. ...

[0069] For example, FIG. 5 Updates to some embodiments of this disclosure and from [other sources] are shown. FIG. 4 The example model is associated with the example model card 402. FIG. 5 In the example, the model may have undergone one or more updates, such as updating the model's name, further training the model using an updated dataset, further training the model to perform one or more new tasks, and / or any other updates. Therefore, by performing at least a portion of process 150, an updated model card 502 can be generated, which includes updating at least the information 406(2) associated with the second field 404(2) of model card 402 to include new information 504(1) and updating the information 406(3) associated with the third field 404(3) of model card 402 to include new information 502(2). Thus, process 150 can automatically generate an updated model card 502 using the updated information 504(1)-(2), rather than requiring one or more users to provide input to update model card 402.

[0070] Return to reference FIG. 1BIn some examples, at least a portion of the data associated with process 150 can differ from the data associated with process 100 in order to update a previously generated model card rather than generate a new model card. For example, request data 124 can represent additional and / or alternative queries specific to an update to a model card. For example, if only one or more particular fields of a model card need to be updated based on one or more updates to model 106, request data 124 can represent one or more queries associated with the particular field(s) without including additional queries associated with model 106. As such, chunk data 126 retrieved using request data 124 can represent information chunks related to the particular field(s) of the model card being updated and / or embeddings associated with the information chunks.

[0071] Additionally, since the language model(s) 130 are being used to update a previously generated model card rather than generate a new model card, prompt data 128 can represent different prompts specific to updating a model card. For example, prompt data 128 can represent prompts that cause the language model(s) 130 to update a previously generated model card and / or update one or more particular fields of a previously generated model card.

[0072] As described further herein, in some examples, it can be important to verify that a model card associated with model 106 is accurate since the model card can be used to evaluate model 106. For a first example, one or more users can use a model card to determine whether model 106 is capable of performing one or more tasks and / or determine the capabilities of a computing device required to execute model 106. For a second example, one or more systems can use a model card to determine whether to provide model 106 to one or more computing devices and / or one or more users for execution. Thus, FIG. 1C An example of a process 156 for verifying a model card associated with a model according to some embodiments of the present disclosure is shown.

[0073] As shown, process 156 can be similar to process 150, except that FIG. 1C In some examples, language model(s) 130 are used to verify a model card represented by card data 152 rather than update a model card. For example, process 156 can again include language model(s) 130 processing at least output data 134, chunk data 146, and card data 152, e.g., during a second processing task associated with second call 132(2). In some examples, language model(s) 130 can perform any type of processing, e.g., described herein with respect to process 150, to verify card data 152. FIG. 11A-11CThe processing. Based at least on the processing, process 156 can include language model(s) 130 generating and / or outputting validation data 158. As described herein, validation data 158 can represent whether a model card is validated (e.g., a validation flag), such as when information included in the model card is accurate, or can represent whether a model card is not validated (e.g., a non-validation flag), such as when at least a portion of information included in the model card is inaccurate.

[0074] In some examples, such as when a model card is not validated, validation data 158 can represent additional information associated with validating the model card. For example, validation data 158 can represent one or more indications of one or more fields from the model card for which information is inaccurate, inaccurate information from the model card, and / or updated information that should be included in the model card to make the model card accurate. For a first example, if a field of a model card associated with a name of model 106 is inaccurate, validation data 158 can represent an indication that the field is inaccurate, an inaccurate current name included in the model card, and / or a correct name that should be included in the model card. For a second example, if a field of a model card associated with a computational requirement for executing model 106 is inaccurate, validation data 158 can represent an indication of the inaccurate field, an inaccurate current computational requirement included in the model card, and / or a correct computational requirement that should be included in the model card.

[0075] Similar to the examples of FIG. 1B In some examples, at least a portion of data associated with process 156 and data associated with process 150 can differ in order to validate a model card rather than update a model card. For example, because language model(s) 130 are being used to validate a model card rather than update a model card, prompt data 128 can represent different prompts specific to validating a model card. For example, prompt data 128 can represent prompts that cause language model(s) 130 to validate a model card, validate one or more specific portions of a model card, and / or generate validation data 158 associated with validating a model card.

[0076] As described herein, in some examples, a model card associated with model 106 can be used to perform one or more additional tasks. For example, FIG. 6An example of one or more systems 602 that can perform various tasks using model cards is shown in accordance with some embodiments of the present disclosure. As shown, the system(s) 602 (which can represent and / or be similar to the example computing device 1200 and / or the example data center 1300) can include at least one or more processors 604 (which can represent and / or be similar to one or more central processing units 1206 and / or one or more graphics processing units 1208), one or more communication interfaces 606 (which can represent and / or be similar to one or more communication interfaces 1210), and a memory 608 (which can represent and / or be similar to the memory 1204). However, in other examples, the system(s) 602 can include additional components.

[0077] As shown, the system(s) 602 can store the generation component(s) 102, the LM component(s) 108, the model loader component(s) 110, the model database(s) 112, the extraction component(s) 118, and / or the reference extraction component(s) 138 in the memory 608. Additionally, the system(s) can use the processor(s) 604 to execute the generation component(s) 102, the LM component(s) 108, the model loader component(s) 110, the model database(s) 112, the extraction component(s) 118, and / or the reference extraction component(s) 138 in order to perform at least a portion of the process 100 of FIG. 1A at least a portion of the process 150 of FIG. 1B and / or at least a portion of the process 156 of FIG. 1C For example, the system(s) 602 can be configured to automatically generate model cards, update model cards, and / or validate model cards.

[0078] In some examples, the system(s) 602 can also be configured to perform one or more tasks using model cards. For example, the system(s) 602 can receive query data 610 representing one or more queries from one or more computing devices 612 (e.g., one or more endpoints). In some examples, the queries can be associated with the computing device(s) 612 seeking information included in a model card associated with a model, such as training details, risk scores, bias details, hardware specifications for optimal performance, etc. Accordingly, based on receiving such queries, the system(s) can obtain a model card from the model database(s) 112 and send card data 614 representing the model card to the computing device(s) 612.

[0079] Additionally or alternatively, in some examples, the system(s) 602 can also enforce execution of the model(s) based at least on criteria for the model card check. In this way, the system(s) 602 can prevent the model(s) from executing in scenarios where, for example, the model(s) are non-compliant within the constraints of the enterprise, are not optimized for the execution environment, and / or the like. Enforcing execution of the model(s) at runtime can enable users and / or organizations to limit execution of the model(s) based at least on factors such as permissions, training data, risk assessment, bias, and / or the like.

[0080] For example, a query received from the computing device(s) 612 can include a request to execute one or more particular models. Accordingly, the system(s) 602 can obtain at least the model card(s) stored in association with the particular model(s), and evaluate the model card(s) relative to one or more criteria associated with the computing device(s) 612. In some examples, the criteria can include a policy (e.g., an enterprise policy, a device policy, a group policy, and / or the like) associated with the computing device(s) 612 that indicates various requirements, expectations, limitations, and / or the like associated with the model(s) that are permitted for use in accordance with the policy. As an example, the policy can indicate, among other things, a risk threshold for the model, a permission requirement for the model, a training requirement for the model, and / or the like. Additionally or alternatively, the criteria can include a hardware specification that indicates one or more limitations and / or capabilities associated with the computing device(s) 612 that will execute the model(s). For example, the hardware specification can indicate features associated with one or more processors of the computing device(s) 612 (e.g., a type of processor, a brand of processor, a model number of processor, and / or the like), memory limitations and / or capabilities associated with the computing device(s) 612, a version number associated with the computing device(s) 612, and / or the like.

[0081] Based at least on the evaluation, the system(s) 602 can determine that the computing device(s) 612 are permitted and / or capable of executing the requested particular model(s). For example, the system(s) 602 can determine that the particular model(s) comply with a given set of requirements (e.g., that can be indicated in the policy), that the particular model(s) are optimized for the execution environment of the computing device(s) 612, and / or that the hardware of the computing device(s) 612 is capable of properly executing the particular model(s). The system(s) 602 can then cause the model data 616 to be transmitted to the computing device(s) 612 for execution of the particular model(s) on the computing device(s) 612.

[0082] However, if the system(s) determine that the computing device(s) 612 are blocked from executing the particular model(s), the system(s) 602 can send an indication to the computing device(s) 612. In some examples, the indication can indicate one or more reasons why the particular model(s) are blocked from being executed on the computing device(s) 612. For example, the indication can indicate that a policy restricts the computing device(s) 612 from executing the particular model(s), and / or the capabilities / limitations of the computing device(s) 612 can block the particular model(s) from being executed.

[0083] In some examples, and as described herein, the model card(s) can indicate risk score(s) associated with the particular model(s), and the system(s) 602 can evaluate the risk score(s) relative to a threshold risk score (e.g., indicated in a policy) associated with the computing device(s) 612. Based at least on the evaluation, the system(s) 602 can determine whether to provide the model data 616 to the computing device(s) 612 to execute the particular model(s). That is, if the risk score(s) of the particular model(s) meet or exceed the risk threshold, the system(s) 602 can determine to block the particular model(s) from being executed on the computing device(s) 612, but if the risk score(s) are less than the risk threshold, the system(s) 602 can determine to allow the particular model(s) to be executed by the computing device(s) 612.

[0084] As another example, the system(s) 602 can determine, based at least on the model card(s), one or more hardware thresholds corresponding to one or more hardware capabilities for executing the particular model(s). The system(s) 602 can then evaluate actual capabilities associated with the computing device(s) 612 relative to the hardware threshold(s) to determine whether to provide the model data 616 to the computing device(s) 612 to execute the particular model(s). If the system(s) 602 determine that the actual capabilities meet or exceed the hardware threshold(s), the system(s) 602 can determine to provide the particular model(s) to the computing device(s) 612, but if the actual capabilities do not meet the hardware threshold(s), the system(s) 602 can determine to block the particular model(s) from being executed by the computing device(s) 612.

[0085] In some examples, the system(s) 602 can propose one or more alternative (e.g., more suitable, more capable, and the like) models to the computing device(s) 612. In some examples, the alternative model(s) can be proposed to the computing device(s) 612 based at least on a determination that the computing device(s) 612 are prohibited from executing the particular model(s). Additionally or alternatively, the computing device(s) 612 can query the system(s) 602 for models that meet certain criteria, prerequisites, intended purposes, and the like. As an example and not by way of limitation, the computing device(s) 612 can request a model for detecting objects in a machine’s environment that has been trained using a closed-source (e.g., non-open source) dataset and optimized for a rural environment. Based on this request, the system(s) 602 can evaluate model cards for various potential model(s) that would meet these requirements.

[0086] While examples herein illustrate the generation component(s) 102, the LM component(s) 108, the model loader component(s) 110, the extraction component(s) 118, and the reference extraction component(s) 138 as including separate components, in other examples, one or more of the generation component(s) 102, the LM component(s) 108, the model loader component(s) 110, the extraction component(s) 118, and the reference extraction component(s) 138 can be combined. Additionally, components can include, without limitation, systems, servers, computing devices, hardware, software, machine learning models, neural networks, transformers, encoders, modules, and / or any other type of processing component configured to perform at least a portion of the processes described herein.

[0087] Referring now to FIG. 7 to FIG. 10 The blocks of the methods 700, 800, 900, and 1000 described herein each include a computational process that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The methods 700, 800, 900, and 1000 can also be embodied as computer-usable instructions stored on computer storage media (e.g., both memory and storage-based media). To this end, the computer-usable or computer-readable storage medium can include a computer data signal embodied in a carrier wave, a computer program product, a memory, a disk, tape, etc., or a combination thereof. By way of example, the computer-readable storage medium can include a physical computer-readable storage medium and / or a physical computer storage medium. These computer-readable storage media can store computer-readable instructions that implement one or more of the methods 700, 800, 900, and 1000. FIG. 1A-1C and FIG. 6 The methods 700, 800, 900, and 1000 are each described in the general context of a method, but the methods 700, 800, 900, and 1000 can additionally or alternatively be embodied as separate software or firmware modules and / or computer programs, procedures, and / or functions. The methods 700, 800, 900, and 1000 can also be implemented more typically as a single closed application, in a distributed computing context, with various features described herein implemented in separate software modules and / or across several computers, storage media, and servers.

[0088] FIG. 7 A flowchart illustrating a method 700 for generating a new model card associated with a model in accordance with some embodiments of the present disclosure is shown. At block B702, the method 700 can include obtaining first information associated with a model. For example, the model loader component(s) 110 can retrieve the first information, such as the source code 114 and / or the document(s) 116 associated with the model 106, from the model database(s) 112. In some examples, the model loader component(s) 110 can then use the extraction component(s) 118 to extract at least a portion of the first information relevant to generating a model card associated with the model 106. For example, the model loader component(s) 110 can use the extraction component(s) 118 to extract at least a portion of the first information associated with one or more queries relevant to generating a model card.

[0089] At block B704, the method 700 can include obtaining a template representing a format for generating a model card. For example, the model loader component(s) 110 can obtain the template data 136 representing a model card template. As described herein, the model card template can represent the fields to be included in a model card and / or the layout of the fields within a model card. In some examples, the model card template can be generic for all model cards, while in other examples, the model card template can be specific to a type of model card and / or a type of model 106.

[0090] At block B706, the method 700 can include generating, based at least on processing input data associated with at least a portion of the first information and the template by one or more language models, output data representing a model card including the format and second information associated with the model. For example, the language model(s) 130 can process the input data associated with at least a portion of the first information and the template. In some examples, the input data to the language model(s) 130 can represent text associated with at least a portion of the first information and the template, while in some examples, the input data can represent one or more embeddings associated with at least a portion of the first information and the template. The language model(s) 130 can then generate card data 148 representing a model card including the format and second information associated with the model 106.

[0091] FIG. 8A flowchart showing a method 800 for updating a model card associated with a model according to some embodiments of the present disclosure is shown. At block B802, the method 800 can include obtaining information associated with a model. For example, the model loader component(s) 110 can retrieve information, such as source code 114 and / or document(s) 116 associated with the model 106, from the model database(s) 112. In some examples, the model loader component(s) 110 can then use the extraction component(s) 118 to extract at least a portion of the information relevant to updating a model card associated with the model 106. For example, the model loader component(s) 110 can use the extraction component(s) 118 to extract at least a portion of the information associated with one or more queries relevant to updating the model card.

[0092] At block B804, the method 800 can include obtaining a model card associated with the model. For example, the model loader component(s) 110 can obtain card data 152 representing a model card associated with the model 106. As described herein, in some examples, the model card can be associated with a previous version of the model 106. For example, the model 106 can have undergone one or more updates, such as further training of the model 106 using a new dataset, after the model card was generated. As such, the model card can no longer represent accurate information associated with the model 106 that is updated.

[0093] At block B806, the method 800 can include generating, based at least on the one or more language models processing input data associated with the at least a portion of the information and the model card, output data representing an updated model card associated with the model. For example, the language model(s) 130 can process input data associated with the at least a portion of the information and the model card. In some examples, the input data of the language model(s) 130 can represent text associated with the at least a portion of the information and the model card, while in some examples, the input data can represent one or more embeddings associated with the at least a portion of the information and the model card. The language model(s) 130 can then generate updated card data 154 representing an updated model card associated with the model 106. For example, the updated model card can include new information representing one or more updates associated with the model 106.

[0094] FIG. 9A flowchart showing a method 900 for validating a model card associated with a model according to some embodiments of the present disclosure is shown. At block B902, the method 900 can include obtaining information associated with a model. For example, the model loader component(s) 110 can retrieve information, such as source code 114 and / or document(s) 116 associated with the model 106, from the model database(s) 112. In some examples, the model loader component(s) 110 can then use the extraction component(s) 118 to extract at least a portion of the information related to updating a model card associated with the model 106. For example, the model loader component(s) 110 can use the extraction component(s) 118 to extract at least a portion of the information associated with one or more queries related to validating a model card.

[0095] At block B904, the method 900 can include obtaining a model card associated with the model. For example, the model loader component(s) 110 can obtain card data 152 representing a model card associated with the model 106. As described herein, in some examples, the model card can be associated with a current version of the model 106. For example, the model card can need to represent current information associated with the model 106.

[0096] At block B906, the method 900 can include generating, based at least on the one or more language models processing input data associated with the at least a portion of the information and the model card, output data indicating whether the model card is validated. For example, the language model(s) 130 can process input data associated with the at least a portion of the information and the model card. In some examples, the input data of the language model(s) 130 can represent text associated with the at least a portion of the information and the model card, while in some examples, the input data can represent one or more embeddings associated with the at least a portion of the information and the model card. The language model(s) 130 can then generate validation data 158 indicating whether the model card is validated. As described herein, for example, if the model card is not validated, the validation data 158 can further represent one or more indications of one or more fields from the model card whose information is inaccurate, inaccurate information from the model card, and / or updated information that should be included in the model card to make the model card accurate.

[0097] FIG. 10A flowchart showing a method 1000 for generating a model card that is then used to determine whether to provide a model to one or more computing devices according to some embodiments of the present disclosure is shown. At block B1002, the method 1000 can include generating output data associated with a model card for a model based at least on processing input data associated with first information corresponding to the model by one or more language models. For example, the language model(s) 130 can process input data associated with the first information, such as information representing source code 114, document(s) 116, and / or a previous model card for the model 106. Based at least on the processing, the language model(s) 130 can generate output data associated with a model card, such as card data 148 representing a new model card or updated card data 154 representing an updated model card.

[0098] At block B1004, the method 1000 can include determining to provide the model card to one or more computing devices based at least on the model card and second information associated with the one or more computing devices. For example, the system(s) 602 can receive a query including second information, such as one or more capabilities associated with the computing device(s) 612 and / or one or more criteria for executing the model 106 on the computing device(s) 612. Then, the system(s) 602 can determine to provide the model 106 to the computing device(s) 602 using one or more of the techniques described herein based at least on comparing the model card to the second information.

[0099] At block B1006, the method 1000 can include sending data for executing the model to the one or more computing devices. For example, the system(s) 602 can send the card data 614 to the computing device(s) 612, where the card data 614 allows the computing device(s) 612 to execute the model 106.

[0100] Example Language Models

[0101] In at least some embodiments, language models, such as large language models (LLMs) and / or other types of generative artificial intelligence (AI) can be implemented. These models can be capable of understanding, summarizing, translating, and / or otherwise generating text (e.g., natural language text, code, etc.), images, videos, computer-aided design (CAD) assets, metaverse and / or meta-universe file information (e.g., in USD format), etc. based on context provided in an input prompt or query. In embodiments, these language models can be considered “large” based on the model being trained on large-scale datasets and having an architecture with a large number of learnable network parameters (weights and biases), such as millions or billions of parameters. LLMs / VLMs, etc. can be implemented to summarize textual data, analyze and extract insights from data (e.g., text, images, videos, etc.), and generate new text / images / videos / etc. in a user-specified style, genre, and / or format. In embodiments, the LLMs of the present disclosure can be used for text processing only, while in other embodiments, multi-modal LLMs can be implemented to accept, understand, and / or generate text as well as other types of content, such as images, audio, and / or video. For example, a visual language model (VLM) or more generally a multi-modal language model can be implemented to accept image, video, audio, text, 3D designs (e.g., CAD), and / or other input data types and / or generate or output images, videos, audio, text, 3D designs, and / or other output data types.

[0102] In various embodiments, various types of LLM / VLM, etc. architectures can be implemented. For example, different architectures can be implemented that use different techniques to understand and generate output, e.g., text, audio, video, images, etc. In some embodiments, LLM architectures can be used, e.g., recurrent neural networks (RNNs) or long short-term memory networks (LSTMs), while in other embodiments, transformer architectures (e.g., those that rely on self-attention mechanisms) can be used to understand and recognize relationships between words or tokens. One or more generative processing pipelines that include an LLM can also include one or more diffusion blocks (e.g., denoiser). Language models of the present disclosure can include encoder and / or decoder block(s). For example, for tasks involving language understanding, e.g., classification, sentiment analysis, question answering, and named entity recognition, a discriminative or encoder-only LLM such as BERT (Bidirectional Encoder Representations from Transformers) can be implemented. As another example, for tasks involving language and content generation, e.g., text completion, story generation, and dialogue generation, a generative or decoder-only LLM such as GPT (Generative Pretrained Transformer) can be implemented. An LLM that includes both encoder and decoder components can be implemented, e.g., T5 (Text-to-Text

[0103] In various embodiments, LLMs / VLMs, etc. can be trained using unsupervised learning, where the LLM learns patterns from a large amount of unlabeled text / audio / video / image / etc. data. As a result of the large amount of training, in embodiments, the model can not require task-specific or domain-specific training. An LLM that has been heavily pre-trained with a large amount of unlabeled text data can be referred to as a base model, and the LLM can be good at a variety of tasks, e.g., question answering, summarization, filling in missing information, and translation. Some LLMs can be customized for specific use cases using techniques such as prompt tuning, fine-tuning, retrieval-augmented generation (RAG), adding adapters (e.g., custom neural networks and / or neural network layers that tune or adjust prompts or tokens to bias the language model towards a particular task or domain), and / or using optimization models for other fine-tuning or customization techniques within a specific task and / or domain.

[0104] In some embodiments, LLMs / VLMs, etc. of the present disclosure can be implemented using various model alignment techniques. For example, in some embodiments, guardrails can be implemented to identify improper or undesirable inputs (e.g., prompts) and / or outputs of the model. In some non-limiting embodiments, implemented guardrails can be similar to those described in U.S. Patent Application No. 18,304,341, filed April 20, 2023, the contents of which are hereby incorporated by reference in their entirety. In some embodiments, one or more additional models or layers thereof can be implemented to identify issues with inputs and / or outputs of the model. For example, these “protective” models can be trained to identify inputs and / or outputs that are “safe” or otherwise normal or desirable and / or “unsafe” or otherwise undesirable for a particular application / implementation. As a result, LLMs / VLMs, etc. of the present disclosure are less likely to output language / text / audio, etc. that can be offensive, vulgar, improper, unsafe, out of domain, and / or otherwise undesirable for a particular application / implementation.

[0105] In some embodiments, LLMs / VLMs, etc. can be configured to or have access to or use one or more plugins, application programming interfaces (APIs), databases, data repositories, repositories, etc. For example, for certain tasks or operations that are not well suited for the model, the model can have instructions (e.g., due to training and / or based on instructions in a given prompt) to access one or more plugins (e.g., third-party plugins) to assist in processing the current input. In such examples, where at least a portion of the prompt is related to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs) to retrieve relevant information. As another example, where at least a portion of the response requires mathematical calculations, the model can access one or more mathematical plugins or APIs to assist in solving the problem(s), which can then be used in the output of the model from the plugins and / or APIs. This process can be repeated, for example, recursively, for any number of iterations and using any number of plugins and / or APIs until a response to the input prompt that solves each inquiry / question / request / process / operation, etc. can be generated. In this way, the model(s) can rely not only on its own knowledge from training with large dataset(s), but also on the specialized knowledge or optimized nature of one or more external resources (e.g., APIs, plugins, etc.).

[0106] In some embodiments, multiple language models (e.g., LLMs / VLMs, etc.), multiple instances of the same language model, and / or multiple prompts provided to the same language model or instance of the same language model can be implemented, executed, or accessed (e.g., using one or more plugins, user interfaces, APIs, databases, data repositories, repositories, etc.) to provide output in response to the same query or in response to separate portions of the query. In at least one embodiment, the same input query and prompts (e.g., constraint sets, regulators, etc.) can be provided to multiple language models, e.g., language models having different architectures, language models trained with different (e.g., updated) data corpora, etc. In one or more embodiments, the language models can be different versions of the same base model. In one or more embodiments, at least one language model can be instantiated as multiple agents, e.g., more than one prompt can be provided to constrain, guide, or otherwise influence the style, content, or flavor of the output provided. In one or more example, non-limiting embodiments, the same language model can be required to provide output corresponding to different roles, perspectives, flavors, or having different knowledge bases, etc., as defined by the prompts provided.

[0107] In any of such embodiments, the output of two or more (e.g., each) language model, two or more versions of at least one language model, two or more instantiated agents of at least one language model, and / or two or more prompts provided to at least one language model can be further processed, e.g., aggregated, compared, or filtered, or used to determine (and provide) a consensus response. In one or more embodiments, output (or version, instance, or agent) from one language model can be provided as input to another language model for further processing and / or verification. In one or more embodiments, a language model can be required to generate or otherwise obtain output about an input source material, where the output is associated with the input source material. Such association can include, for example, generating a title or portion of text to be embedded (e.g., as metadata) with the input source text or image. In one or more embodiments, the output of a language model can be used to determine the validity of input source material for further processing or inclusion in a dataset. For example, a language model can be used to assess the presence (or absence) of a target word in a portion of text or an object in an image, where the text or image is annotated to record such presence (or lack thereof). Alternatively, determinations from a language model can be used, for example, but not limited to, to determine whether source material should be included in a curated dataset.

[0108] FIG. 11A is a block diagram of an example generative language model system 1100 suitable for implementing at least some embodiments of the present disclosure. In FIG. 11AIn the illustrated example, the generative language model system 1100 includes a retrieval-augmented generation (RAG) component 1192, an input processor 1105, a tokenizer 1110, an embedding component 1120, a plug-in / API 1195, and a generative language model (LM) 1130 (which can include a LLM, a VLM, a multi-modal LM, etc.).

[0109] At a high level, the input processor 1105 can receive an input 1101, which includes text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasound, etc.), 3D design data, CAD data, Universal Scene Description (USD) data, etc.), depending on the architecture of the generative LM 1130. In some embodiments, the input 1101 includes plaintext in the form of one or more sentences, paragraphs, code snippets, and / or documents. Additionally or alternatively, the input 1101 can include sequences of numbers, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., in the form of a table, JSON, or XML). In some implementations where the generative LM 1130 is capable of processing multi-modal input, the input 1101 can combine text with image data, audio data, and / or other types of input data such as those described herein. Taking the example of raw input text, the input processor 1105 can prepare the raw input text in various ways. For example, the input processor 1105 can perform various types of text filtering to remove noise (e.g., special characters, punctuation, HTML tags, stop words) from the relevant textual content. In examples involving stop words (commonly used words that tend to have little semantic meaning), the input processor 1105 can remove the stop words to reduce noise and focus the generative LM 1130 on more meaningful content. The input processor 1105 can apply text normalization, for example, by converting all characters to lowercase, removing diacritics, and / or handling special cases such as acronyms or abbreviations to ensure consistency. These are just a few examples, and other types of input processing can be applied.

[0110] In some embodiments, the RAG component 1192 can be used to retrieve additional information to be used as part of the input 1101 or prompt. For example, in some embodiments, the input 1101 can be generated using a query or input to a model (e.g., a question, a request, etc.) in addition to data retrieved using the RAG component 1192. In some embodiments, the input processor 1105 can analyze the input 1101 and communicate with the RAG component 1192 (or in embodiments, the RAG component 1192 can be part of the input processor 1105) in order to identify relevant text and / or other data to provide as additional context or information sources to the generative LM 1130, typically from which a response, answer, or output 1190 is identified. For example, where the input indicates that the user is interested in the desired tire pressure for a particular make and model of vehicle, the RAG component 1192 can retrieve tire pressure information or text corresponding thereto from a digital (embedded) version of the user manual for that particular vehicle make and model, e.g., using a vector search in an embedding space. Similarly, where the user revisits a chatbot related to a particular product offering or service, the RAG component 1192 can retrieve a previously stored session history or at least an abstract thereof and include the previous session history along with the current inquiry / request as part of the input 1101 to the generative LM 1130.

[0111] The tokenizer 1110 can segment (e.g., processed) text into smaller units (tokens) for subsequent analysis and processing. Depending on the implementation, tokens can represent individual words, subwords, characters, etc. Word-based tokenization divides text into individual words, treating each word as a separate token. Subword tokenization breaks words into smaller meaningful units (e.g., prefixes, suffixes, stems), enabling the generative LM 1130 to understand morphological variations and more efficiently process out-of-vocabulary words. Character-based tokenization represents each character as a separate token, enabling the generative LM 1130 to process text at a fine-grained level. The choice of tokenization strategy can depend on factors such as the language being processed, the task at hand, and / or characteristics of the training dataset. As such, the tokenizer 1110 can convert (e.g., processed) text into a structured format according to the tokenization mode implemented in a particular embodiment.

[0112] The embedding component 1120 can transform discrete tokens into a (e.g., dense, continuous vector) representation of semantic meaning using any known embedding technique. For example, the embedding component 1120 can use pre-trained word embeddings (e.g., Word2Vec, GloVe, or FastText), one-hot encoding, Term Frequency-Inverse Document Frequency (TF-IDF) encoding, one or more embedding layers of a neural network, and / or other ways.

[0113] In some implementations in which input 1101 includes image data, input processor 1101 can resize image data to a standard size compatible with the format of the corresponding input channel and / or can normalize pixel values to a common range (e.g., 0 to 1) to ensure consistent representation, and embedding component 1120 can encode image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations in which input 1101 includes audio data, input processor 1101 can resample audio files to a consistent sampling rate for uniform processing, and embedding component 1120 can extract and encode audio features using any known technique, e.g., in the form of a spectrogram (e.g., a mel-spectrogram). In some implementations in which input 1101 includes video data, input processor 1101 can extract frames or apply resizing to extracted frames, and embedding component 1120 can extract features such as optical flow embeddings or video embeddings and / or can encode temporal information or sequences of frames. In some implementations in which input 1101 includes multi-modal data, embedding component 1120 can fuse representations of different types of data (e.g., text, image, audio) using techniques such as early fusion (concatenation), late fusion (sequential processing), attention-based fusion, etc.

[0114] Generative LM 1130 and / or other components of generative LLM system 1100 can use different types of neural network architectures depending on the implementation. For example, transformer-based architectures (e.g., such as those used in models of GPT) can be implemented, and the transformer-based architectures can include a self-attention mechanism that weighs the importance of different words or tokens in an input sequence and / or a feed-forward network that processes the output of a self-attention layer, applies a non-linear transformation to the input representation, and extracts higher-level features. Some non-limiting example architectures include transformers (e.g., encoder-decoder, decoder-only, multi-modal), RNNs, LSTMs, fusion models, diffusion models, cross-modal embedding models that learn a joint embedding space, graph neural networks (GNNs), hybrid architectures that combine different types of architectures, adversarial networks such as generative adversarial networks or GANs, or adversarial autoencoders (AAEs) for joint distribution learning, etc. As such, depending on the implementation and architecture, embedding component 1120 can apply the encoded representation of input 1101 to generative LM 1130, and generative LM 1130 can process the encoded representation of input 1101 to generate output 1190, which can include responsive text and / or other types of data.

[0115] As described herein, in some embodiments, generative LM 1130 can be configured to access or use (or be able to access or use) plugins / APIs 1195 (which can include one or more plugins, application programming interfaces (APIs), databases, data repositories, repositories, etc.). For example, for certain tasks or operations that are not well suited for generative LM 1130, the model can have instructions (e.g., retrieved using RAG component 1192, such as those instructions in a given prompt) to access one or more plugins / APIs 1195 (e.g., third-party plugins) to help process the current input. In such examples, where at least a portion of the prompt is related to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs), send at least a portion of the prompt related to the particular plugin / API 1195 to the plugin / API 1195, which can process the information and return an answer to generative LM 1130, and generative LM 1130 can use the response to generate output 1190. This process can be repeated, for example, recursively, for any number of iterations and using any number of plugins and / or APIs 1195 until output 1190 can be generated that addresses each query / question / request / process / operation, etc. from input 1101. In this way, the model(s) can rely not only on its own knowledge from training with large dataset(s) and / or from data retrieved using RAG component 1192, but also on the specialized knowledge or optimized properties of one or more external resources (e.g., plugins / APIs 1195).

[0116] FIG. 11B is a block diagram of an example implementation of generative LM 1130 including a transformer encoder-decoder. For example, assume that an input text such as “Who discovered gravity?” (e.g., tokenized by tokenizer 1110 of FIG. 11A is tokenized into tokens such as words, and each token (e.g., encoded by embedding component 1120 of FIG. 11A is encoded into a corresponding embedding (e.g., of size 512). Since these token embeddings do not typically represent the position of the token in the input sequence, position encodings can be added to each token embedding using any known technique to encode the sequential relationship and context of the tokens in the input sequence. In this way, the (e.g., resulting) embeddings can be applied to one or more encoders 1135 of generative LM 1130.

[0117] In example implementations, the encoder(s) 1135 form an encoder stack, where each encoder includes a self-attention layer and a feed-forward network. In an example transducer architecture, each token (e.g., word) flows through a separate path. As such, each encoder can accept a sequence of vectors, pass each vector through a self-attention layer, then through a feed-forward network, and then up to the next encoder in the stack. Any known self-attention technique can be used. For example, to compute a self-attention score for each token (word), a query vector, a key vector, and a value vector can be created for each token, a self-attention score for a pair of tokens can be computed by taking the dot product of the query vector and the corresponding key vector, normalizing the resulting score, multiplying by the corresponding value vector, and summing the weighted value vectors. The encoder can apply multi-head attention, where the attention mechanism is applied multiple times in parallel with different learned weight matrices. Any number of encoders can be concatenated to generate a context vector that encodes the input. The attention projection layer 1140 can convert the context vector into attention vectors (keys and values) for the decoder(s) 1145.

[0118] In example implementations, the decoder(s) 1145 form a decoder stack, where each decoder includes a self-attention layer, an encoder-decoder self-attention layer that focuses on relevant portions of the input sequence using attention vectors (keys and values) from the encoders, and a feed-forward network. Like the encoder(s) 1135, in an example transducer architecture, each token (e.g., word) flows through a separate path in the decoder(s) 1145. During a first pass, the decoder(s) 1145, the classifier 1150, and the generator 1155 can generate a first token, and the generator 1155 can apply the generated token as input during a second pass. This process can repeat in a loop, successively generating and adding tokens (e.g., words) to the output of the previous pass, and applying the token embedding of the composite sequence with positional encodings as input to the decoder(s) 1145 during the next pass, one token at a time in sequence (known as autoregressive), until a symbol or token is predicted that indicates the end of the response. Within each decoder, the self-attention layer is typically constrained to only attend to previous positions in the output sequence by applying a masking technique (e.g., setting future positions to negative infinity) before a normalization exponential function (softmax) operation. In example implementations, the encoder-decoder attention layer operates similarly to the (e.g., multi-head) self-attention in the encoder(s) 1135, except that it creates its queries from the layer below it, and takes its keys and values (e.g., matrices) from the output of the encoder(s) 1135.

[0119] In this way, the decoders 1145 can output some decoded (e.g., vector) representations of the inputs applied during a particular pass. The classifier 1150 can include a multi-class classifier comprising one or more neural network layers that project the decoded (e.g., vector) representations onto corresponding dimensions (e.g., one dimension for each supported word or token in the output vocabulary), and a normalized exponential function (softmax) operation that converts logits into probabilities. The generation mechanism 1155 can then select or sample words or tokens based on their corresponding predicted probabilities (e.g., selecting the word with the highest predicted probability) and append them to the output of the previous pass, sequentially generating each word or token. The generation mechanism 1155 can repeat this process, triggering successive decoder inputs and corresponding predictions until a symbol or token indicating the end of the response is selected or sampled, at which point the generation mechanism 1155 can output the generated response.

[0120] FIG. 11C This is a block diagram of an exemplary implementation of the generative LM 1130, including a decoder-only converter architecture. For example, FIG. 11C The (multiple) decoders 1160 can be used with FIG. 11B The (multiple) decoders 1145 operate similarly, the difference being... FIG. 11C Each of the (multiple) decoders 1160 omits the encoder-decoder self-attention layer (because there is no encoder in this embodiment). In this way, the (multiple) decoders 1160 can form a decoder stack, where each decoder includes a self-attention layer and a feedforward network. Furthermore, instead of encoding the input sequence, a symbol or tag indicating the end of the input sequence (or the beginning of the output sequence) can be appended to the input sequence, and the resulting sequence (e.g., a corresponding embedding with positional encoding) can be applied to the (multiple) decoders 1160. FIG. 11B Similar to the (multiple) decoders 1145, each tag (e.g., a word) can flow through a separate path in the (multiple) decoders 1160, and the (multiple) decoders 1160, classifier 1165, and generator 1170 can use autoregression to sequentially generate one tag at a time until a symbol or tag indicating the end of the response is predicted. Classifier 1165 and generator 1170 can be... FIG. 11BThe classifier 1150 and generator 1155 operate similarly, wherein the generator 1170 selects each consecutive output label based on the corresponding predicted probability or samples each consecutive output label and appends it to the output of the previous iteration, generating each label sequentially until a symbol or label indicating the end of the response is selected or sampled. These and other architectures described herein are merely examples, and other suitable architectures may be implemented within the scope of this disclosure.

[0121] Exemplary computing device

[0122] FIG. 12 This is a block diagram of one or more exemplary computing devices 1200 suitable for implementing some embodiments of the present disclosure. The computing device 1200 may include an interconnect system 1202 directly or indirectly coupled to: a memory 1204, one or more central processing units (CPUs) 1206, one or more graphics processing units (GPUs) 1208, a communication interface 1210, input / output (I / O) ports 1212, input / output components 1214, a power supply 1216, one or more presentation components 1218 (e.g., multiple displays), and one or more logic units 1220. In at least one embodiment, the computing device 1200 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more of the GPUs 1208 may include one or more vGPUs, one or more of the CPUs 1206 may include one or more vCPUs, and / or one or more of the logic units 1220 may include one or more virtual logic units. In this way, (multiple) computing devices 1200 may include discrete components (e.g., an entire GPU dedicated to computing device 1200), virtual components (e.g., a portion of the GPU dedicated to computing device 1200), or a combination thereof.

[0123] although FIG. 12 The various blocks are shown as being connected to lines via interconnect system 1202, but this is not intended to be limiting, but only for clarity. For example, in some embodiments, a presentation component 1218 such as a display device can be considered as I / O component 1214 (e.g., if the display is a touchscreen). As another example, CPU 1206 and / or GPU 1208 may include memory (e.g., memory 1204 may represent a storage device other than the memory of GPU 1208, CPU 1206, and / or other components). In other words, FIG. 12The computing device 1200 is merely illustrative. Distinction is not made between "workstation" "server", "laptop", "desktop", "tablet", "client device", "mobile device", "hand-held device", "game console", "electronic control unit (ECU)", "virtual reality system", and / or other device or system types within the scope of what is FIG. 12

[0124] The interconnection between elements and / or interconnection between a element and a process can be implemented by any hardware means and / or software means. For example, a communication between elements can be facilitated by one or more buses. The interconnection between elements and / or the interconnection between an element and a process can also be implemented by a communication means, such as a wireless communication, a wired communication, and / or the like. The interconnection between elements and / or the interconnection between an element and a process can also be olded by a remote communication, such as via the Internet, a small area network, and / or the like.

[0125] The memory 1204 can include any of a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computing device 1200. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media. Computer storage media can include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1204 can store computer-readable instructions such as those representing a program and / or program elements such as an operating system. Computer storage media can further include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by computing device 1200. Computer storage media, as used herein, does not include signals per se.

[0126]

[0127] ​​Computer storage media can include computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term "modulated data signal" can refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of the any of the above can also be included within the scope of computer readable media.

[0128] The CPU(s) 1206 can be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1200 to perform one or more of the methods and / or processes described herein. The CPU(s) 1206 can each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of concurrently processing multiple software threads. The CPU(s) 1206 can include any type of processors and can include different types of processors depending on the type of computing device 1200 being implemented (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of computing device 1200, the processor can be an Advanced RISC Machines (ARM) processor implemented using a Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 1200 can include one or more CPUs 1206 in addition to one or more microprocessors or auxiliary processor cores (e.g., a math co-processor).

[0129] In addition to, or instead of, the CPU(s) 1206, the GPU(s) 1208 can be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1200 to perform one or more of the methods and / or processes described herein. One or more of the GPU(s) 1208 can be integrated GPUs (e.g., integrated with one or more of the CPU(s) 1206) and / or one or more of the GPU(s) 1208 can be discrete GPUs. In embodiments, one or more of the GPU(s) 1208 can be a co-processor of one or more of the CPU(s) 1206. The GPU(s) 1208 can be used by the computing device 1200 to render graphics (e.g., 3D graphics) or to perform general-purpose computing. For example, the GPU(s) 1208 can be used for general-purpose computing on GPUs (GPGPU). The GPU(s) 1208 can include hundreds or thousands of cores capable of processing hundreds or thousands of software threads concurrently. The GPU(s) 1208 can generate pixel data for output images in response to rendering commands (e.g., rendering commands received from the CPU(s) 1206 via a host interface). The GPU(s) 1208 can include graphics memory, such as display memory, for storing pixel data or any other suitable data (e.g., GPGPU data). The display memory can be included as part of the memory 1204. The GPU(s) 1208 can include two or more GPUs operating in parallel (e.g., via a link). The link can connect the GPUs directly (e.g., using NVLINK) or can connect the GPUs through a switch (e.g., using an NVSwitch). When grouped together, each GPU 1208 can generate pixel data or GPGPU data for different portions of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can include its own memory or can share memory with other GPUs.

[0130] In addition to, or instead of, the CPU(s) 1206 and / or GPU(s) 1208, the logic unit(s) 1220 can be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1200 to perform one or more of the methods and / or processes described herein. In embodiments, the CPU(s) 1206, GPU(s) 1208, and / or logic unit(s) 1220 can perform any combination of methods, processes, and / or portions thereof, discretely or jointly. One or more of the logic units 1220 can be part of and / or integrated within one or more of the CPU(s) 1206 and / or GPU(s) 1208, and / or one or more of the logic units 1220 can be discrete components or otherwise external to the CPU(s) 1206 and / or GPU(s) 1208. In embodiments, one or more of the logic units 1220 can be a co-processor of one or more of the CPU(s) 1206 and / or one or more of the GPU(s) 1208.

[0131] Examples of the logic unit(s) 1220 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), a Tensor Core (TC), a tensor processing unit (TPU), a pixel visual core (PVC), a visual processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multi-processor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application-specific integrated circuit (ASIC), a floating-point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) element, and / or the like.

[0132] The communication interface 1210 can include one or more receivers, transmitters, and / or transceivers that enable the computing device 1200 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communication. The communication interface 1210 can include components and functionality to enable communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or InfiniBand), low power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit(s) 1220 and / or the communication interface 1210 can include one or more data processing units (DPUs) for sending data received over a network and / or over the interconnect system 1202 directly to the memory of one or more GPUs 1208, for example.

[0133] The I / O ports 1212 can enable the computing device 1200 to be logically coupled to other devices including the I / O components 1214, the presentation component(s) 1218, and / or other components, some of which can be built into (e.g., integrated with) the computing device 1200. Exemplary I / O components 1214 include a microphone, a mouse, a keyboard, a joystick, a game pad, a game controller, a satellite antenna, a scanner, a printer, a wireless device, etc. The I / O components 1214 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some cases, inputs can be transmitted to an appropriate network element for further processing. A NUI can implement any combination of speech recognition, touch recognition, facial recognition, biometric recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, and touch recognition associated with a display of the computing device 1200, as described in more detail below. The computing device 1200 can include a depth camera, such as a stereoscopic camera system, an infrared camera system, an RGB camera system, touch screen technology, and combinations of these, for gesture detection and recognition. In addition, the computing device 1200 can include an accelerometer or a gyroscope (e.g., as part of an inertial measurement unit (IMU)) that is capable of detecting motion. In some examples, the computing device 1200 can use the output of the accelerometer or gyroscope to render an immersive augmented reality or virtual reality.

[0134] The power supply 1216 can include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply 1216 can provide power to the computing device 1200 to enable the components of the computing device 1200 to operate.

[0135] The multiple presentation units 1218 may include displays (e.g., monitors, touchscreens, television screens, head-up displays (HUDs), other display types, or combinations thereof), speakers, and / or other presentation units. The multiple presentation units 1218 may receive data from other units (e.g., multiple GPUs 1208, multiple CPUs 1206, DPUs, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0136] Exemplary Data Center

[0137] FIG. 13 An exemplary data center 1300 that can be used in at least one embodiment of this disclosure is shown. The data center 1300 may include a data center infrastructure layer 1310, a framework layer 1320, a software layer 1330, and / or an application layer 1340.

[0138] like FIG. 13 As shown, the data center infrastructure layer 1310 may include a resource orchestrator 1312, packet computing resources 1314, and node computing resources (“nodes CR”) 1316(1)-1316(N), where “N” represents any complete positive integer. In at least one embodiment, nodes CR 1316(1)-1316(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules and / or cooling modules, etc. In some embodiments, one or more node CRs from nodes CR 1316(1)-1316(N) may correspond to servers having one or more of the aforementioned computing resources. Additionally, in some embodiments, nodes CR1316(1)-1316(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more of nodes CR1316(1)-1316(N) may correspond to virtual machines (VMs).

[0139] In at least one embodiment, the packet computing resource 1314 may include individual packets of nodes CR1316 housed within one or more racks (not shown), or a plurality of racks housed in data centers at various geographical locations (also not shown). Individual packets of nodes CR1316 within the packet computing resource 1314 may include packet computing, networking, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several nodes CR1316, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any combination of any number of power modules, cooling modules, and / or network switches.

[0140] Resource orchestrator 1312 can be configured or otherwise controlled to control one or more nodes CR1316(1)-1316(N) and / or grouped computing resources 1314. In at least one embodiment, resource orchestrator 1312 may include a Software Design Infrastructure (SDI) management entity for data center 1300. Resource orchestrator 1312 may include hardware, software, or some combination thereof.

[0141] In at least one embodiment, such as FIG. 13 As shown, framework layer 1320 may include a job scheduler 1328, a configuration manager 1334, a resource manager 1336, and / or a distributed file system 1338. Framework layer 1320 may include a framework to support software 1332 of software layer 1330 and / or one or more applications 1342 of application layer 1340. Software 1332 or (multiple) applications 1342 may respectively include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 1320 may be, but is not limited to, a free and open-source software web application framework, such as Apache Spark. TM(“Spark”), which can leverage the distributed file system 1338 for large-scale data processing (e.g., “big data”). In at least one embodiment, the job scheduler 1328 can include a Spark driver program to facilitate scheduling of workloads supported by various tiers of the data center 1300. The configuration manager 1334 can be capable of configuring different tiers, such as the software tier 1330 and the framework tier 1320 including Spark and the distributed file system 1338, to support large-scale data processing. The resource manager 1336 can be capable of managing clustered or grouped computing resources mapped to or allocated for supporting the distributed file system 1338 and the job scheduler 1328. In at least one embodiment, clustered or grouped computing resources can include grouped computing resources 1314 at data center infrastructure layer 1310. The resource manager 1336 can coordinate with the resource orchestrator 1312 to manage these mapped or allocated computing resources.

[0142] In at least one embodiment, software 1332 included in software tier 1330 can include software used by at least portions of node C.R.s 1316(1)-1316(N), grouped computing resources 1314, and / or distributed file system 1338 of framework tier 1320. One or more types of software can include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0143] In at least one embodiment, application(s) 1342 included in application tier 1340 can include one or more types of applications used by at least portions of node C.R.s 1316(1)-1316(N), grouped computing resources 1314, and / or distributed file system 1338 of framework tier 1320. One or more types of applications can include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0144] In at least one embodiment, any of configuration manager 1334, resource manager 1336, and resource orchestrator 1312 can implement any number and type of self-modifying actions based on any amount and type of data acquired in any technically feasible manner. Self-modifying actions can mitigate data center operators of data center 1300 making possibly poor configuration decisions, and can avoid underutilization and / or poor performance of portions of the data center.

[0145] Data center 1300 may include tools, services, software, or other resources to train one or more machine learning models, or to use one or more machine learning models to predict or infer information according to one or more embodiments described herein. For example, machine learning models(s) can be trained by computing weight parameters based on a neural network architecture using the software and / or computing resources described above with respect to data center 1300. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks can be used to infer or predict information using resources described above with respect to data center 1300 by using weight parameters computed through one or more training techniques (e.g., but not limited to those described herein).

[0146] In at least one embodiment, the data center 1300 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as services to allow a user to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services.

[0147] Exemplary network environment

[0148] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, networked storage devices (NAS), other back-end devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... FIG. 12 Implemented on one or more instances of computing devices 1200, for example, each device may include similar components, features, and / or functions of computing devices 1200. Additionally, in the case of implementing back-end devices (e.g., servers, NAS, etc.), the back-end devices may be included as part of data center 1300, examples of which are described herein. FIG. 13 To describe in more detail.

[0149] Components of a network environment can communicate with each other via networks, which can be wired, wireless, or a combination of both. A network may include multiple networks or one of several networks. For example, a network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0150] A compatible network environment can include one or more peer-to-peer network environments, in which case no servers can be included in the network environment, and one or more client-server network environments, in which case one or more servers can be included in the network environment. In a peer-to-peer network environment, functionality described herein with respect to server(s) can be implemented on any number of client devices.

[0151] In at least one embodiment, a network environment can include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment can include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which can include one or more core network servers and / or edge servers. The framework layer can include a framework that supports a software layer and / or one or more applications of an application layer. The software or application(s) can include web-based service software or applications, respectively. In an embodiment, one or more client devices can use the web-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer can be, without limitation, a free and open-source software web application framework, such as Apache® Hadoop®, which can use the distributed file system for large-scale data processing (e.g., “big data”).

[0152] A cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functionality described herein (or one or more portions thereof). Any of these various functionalities can be distributed across multiple locations beyond a central or core server (e.g., possibly across one or more data centers distributed across states, regions, countries, the Earth, and the like). Core server(s) can designate at least a portion of the functionality to edge server(s) if the connection to the user (e.g., client device) is relatively close to the edge server(s). A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available for use by many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0153] Client device(s) can include any of the client devices described herein with respect to FIG. 12At least some of the components, features and functionality of the example computing device 1200 described above can be implemented as software modules, hardware modules or other module. As used herein the term "module" or "component" can include a computer-related entity, be it software, hardware, a combination of hardware and software, or firmware, or any other entity, that is configured to perform certain operations. The modules can also be implemented as a hardware device, software device, or a combination of hardware and software. In a hardware implementation, for example, a processing device can include one or more physical processors (e.g., central processing units (CPUs), microprocessors, microcontrollers, digital signal processors, complex programmable logic devices (CPLDs), programmable logic devices (PLDs), etc.). A processing device can execute software modules or computer-executable instructions to perform certain operations, tasks, or functions. In some embodiments, the software modules are stored in the system memory 1202, disk drive, or other computer- readable media. The software modules or instructions can be executed by one or more processors or processing devices.

[0154] The present disclosure can be described in the general context of computer code or machine-useable instructions, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules including routines, programs, objects, components, data structures, and the like, refer to code that performs particular tasks or implements particular abstract data types. The present disclosure can be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general- purpose computers, more specialty computing devices, and the like. The present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0155] As used herein, including the claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a" or "the" component includes a plurality of such components unless the context clearly indicates otherwise. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. As used herein, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless specified otherwise, or clear from the context, the phrase "X employs A or B" is intended to mean that X employs A or B or both. Also, the term "coupled" as used herein is intended to mean one or more mechanical, electrical, and / or magnetic connection(s) between the items connected. As used herein, the term "associated with" is intended to mean that the associated items are in some manner connected, either directly or indirectly, or are in some manner not connected, but are deemed to be connected for purposes of the present disclosure. As used herein, the term "and / or" is intended to mean either "and" or "or." As used herein, the term "about" means approximately or nearly, for example, the term "about 5" means approximately 5. As used herein, the term "comprises" means includes, but not limited to.

[0156] The subject matter of the present disclosure is described herein with specificity to meet statutory requirements. However, the description itself is not intended to limit the scope of the disclosure. Rather, the inventors have contemplated that the claimed subject matter can also be embodied in other ways, to include different steps or combinations of steps in conjunction with the ones described in this document, including those not expressly described herein. Additionally, although the terms “step” and / or “block” can be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.

[0157] Exemplary Paragraph

[0158] A: A method comprising: generating, based at least on one or more language models processing input data representing first information associated with a machine learning model, output data representing a model card, the model card comprising second information describing the machine learning model; determining, based at least on the model card or a template and one or more capabilities associated with one or more computing devices, to provide the machine learning model to the one or more computing devices; and sending, to the one or more computing devices, data for executing the machine learning model.

[0159] B: The method of paragraph A, further comprising: obtaining one or more queries associated with one or more fields included in the model card; and extracting, based at least on the one or more queries, the first information from at least one of source code associated with the machine learning model, one or more documents describing the machine learning model, or a second model card associated with the machine learning model.

[0160] C: The method of paragraph A or paragraph B, further comprising: obtaining a template, the template comprising a format for generating the model card, wherein: generating the model card is further based at least on the one or more language models processing second input data representing the template; and the model card comprises the second information arranged according to the format from the template.

[0161] D: The method of any of paragraphs A-C, further comprising: obtaining a second model card associated with the machine learning model, the second model card comprising third information describing the machine learning model, wherein: generating the model card is further based at least on the one or more language models processing second input data representing the second model card; and at least a portion of the second information included in the model card comprises updated information compared to the third information included in the second model card.

[0162] E: The method of any of paragraphs A-D, wherein generating the model card comprises: generating initial output data based at least on processing the input data based on the one or more language models; and generating the output data representing the model card based at least on processing the initial output data and second input data representing at least one of a template associated with the model card or a second model card associated with the machine learning model based on the one or more language models.

[0163] F: The method of any of paragraphs A-E, further comprising: obtaining third information associated with one or more second machine learning models, wherein generating the model card is further based at least on processing second input data representing the third information based on the one or more language models.

[0164] G: The method of paragraph F, wherein obtaining the second information comprises extracting the second information based at least on the first information from at least one of source code associated with the one or more second machine learning models, one or more documents associated with the one or more second machine learning models, or one or more model cards associated with the one or more second machine learning models.

[0165] H: The method of any of paragraphs A-G, further comprising: retrieving one or more embeddings associated with the first information from one or more databases, wherein the input data representing the first information comprises at least the one or more embeddings.

[0166] I: The method of any of paragraphs A-H, wherein the second information comprises at least one of: an identifier associated with the machine learning model; one or more identifiers of one or more data sets used to train the machine learning model; one or more sizes of the one or more data sets; one or more license types associated with the machine learning model; one or more risk scores associated with the machine learning model; one or more bias scores associated with the machine learning model; one or more inputs of the machine learning model; one or more outputs from the machine learning model; one or more intended users associated with the machine learning model; or one or more computational requirements associated with executing the machine learning model.

[0167] J: A system comprising: one or more processors to: obtain, from one or more databases, first information corresponding to a machine learning model; generate, based at least on one or more language models, output data representing a model card comprising second information describing the machine learning model, the output data generated based on processing input data associated with the first information; and perform, based at least on the model card, one or more operations associated with the machine learning model.

[0168] K: The system of paragraph J, wherein the first information is obtained at least by: obtaining one or more queries associated with one or more fields included in the model card; generating one or more first embeddings associated with the one or more queries; and retrieving, from the one or more databases, one or more second embeddings related to the one or more first embeddings, the one or more second embeddings associated with the first information.

[0169] L: The system of paragraph J or paragraph K, wherein the one or more processors are further to: obtain a template comprising a format for generating the model card, wherein: the model card is further generated based at least on the one or more language models processing second input data representing the template; and the model card comprises the second information arranged according to the format from the template.

[0170] M: The system of any of paragraphs J-L, wherein the one or more processors are further to: obtain a second model card associated with the machine learning model, the second model card comprising third information describing the machine learning model, wherein: the model card is further generated based at least on the one or more language models processing second input data representing the second model card; and at least a portion of the second information included in the model card comprises updated information compared to the third information included in the second model card.

[0171] N: The system of any of paragraphs J-M, wherein generating the model card comprises: generating, based at least on the one or more language models, initial output data based on processing the input data; obtaining second input data representing at least one of a template associated with the model card or a second model card associated with the machine learning model; and generating, based at least on the one or more language models, the output data representing the model card based on processing the initial output data and the second input data.

[0172] O: The system of any of paragraphs J-N, wherein the one or more processors are further to: obtain third information associated with one or more second machine learning models, wherein the model card is further to generate, based at least on the one or more language models, second input data associated with the second information.

[0173] P: The system of paragraph O, wherein the second information is obtained by at least extracting the second information from at least one of source code associated with the one or more second machine learning models, one or more documents associated with the one or more second machine learning models, or one or more model cards associated with the one or more second machine learning models based at least on the first information.

[0174] Q: The system of any of paragraphs J-P, wherein performing the one or more operations comprises at least one of: storing the model card in association with the machine learning model; or determining, based at least on at least one of one or more policies or one or more capabilities associated with one or more computing devices and the model card, whether to provide the model card to the one or more computing devices.

[0175] R: The system of any of paragraphs J-Q, wherein the system is included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing optical transport simulations; a system for collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using edge devices; a system implemented using robots; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more visual language models (VLMs); a system for performing operations using one or more multi-modal language models; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0176] S: One or more processors comprising: processing circuitry to: generate one or more embeddings associated with information describing a machine learning model; generate, based at least on one or more language models, output data representing a model card from input data associated with the one or more embeddings, the model card comprising at least a portion of the information describing the machine learning model; and store the model card in association with the machine learning model.

[0177] T: The one or more processors of paragraph S, wherein the one or more processors are included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing optical transport simulation; a system for collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using edge devices; a system implemented using robots; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more visual language models (VLMs); a system for performing operations using one or more multi-modal language models; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Claims

1. A method comprising: processing, based at least on one or more language models, input data representing first information associated with a machine learning model to generate output data representing a model card, the model card comprising second information describing the machine learning model; determining, based at least on the model card or a template and one or more capabilities associated with one or more computing devices, to provide the machine learning model to the one or more computing devices; and sending, to the one or more computing devices, data for executing the machine learning model.

2. The method of claim 1, further comprising: obtaining one or more queries associated with one or more fields included in the model card; and extracting, based at least on the one or more queries, the first information from at least one of source code associated with the machine learning model, one or more documents describing the machine learning model, or a second model card associated with the machine learning model.

3. The method of claim 1, further comprising: obtaining a template, the template comprising a format for generating the model card, wherein: generating the model card is further based at least on the one or more language models processing second input data representing the template; and the model card comprises the second information arranged according to the format from the template.

4. The method of claim 1, further comprising: obtaining a second model card associated with the machine learning model, the second model card comprising third information describing the machine learning model, wherein: generating the model card is further based at least on the one or more language models processing second input data representing the second model card; and at least a portion of the second information included in the model card comprises updated information compared to the third information included in the second model card. generating the model card comprises: generating initial output data based at least on the one or more language models processing the input data; and 5. The method of claim 1, wherein, generating the output data representing the model card based at least on the one or more language models processing the initial output data and second input data representing at least one of a template associated with the model card or a second model card associated with the machine learning model.

6. The method of claim 1, further comprising: obtaining third information associated with one or more second machine learning models, wherein generating the model card is further based at least on the one or more language models processing second input data representing the third information. obtaining the second information comprises extracting the second information based at least on the first information from at least one of source code associated with the one or more second machine learning models, one or more documents associated with the one or more second machine learning models, or one or more model cards associated with the one or more second machine learning models.

8. The method of claim 1, further comprising:

7. The method of claim 6, wherein, ​ ​ retrieving, from one or more databases, one or more embeddings associated with the first information, wherein the input data representing the first information includes at least the one or more embeddings.

9. The method of claim 1, wherein, The second information includes at least one of: an identifier associated with the machine learning model; one or more identifiers of one or more datasets used to train the machine learning model; one or more sizes of the one or more datasets; one or more license types associated with the machine learning model; one or more risk scores associated with the machine learning model; one or more bias scores associated with the machine learning model; one or more inputs to the machine learning model; one or more outputs from the machine learning model; one or more intended users associated with the machine learning model; or one or more computational requirements associated with executing the machine learning model.

10. A system comprising: one or more processors to: obtain, from one or more databases, first information corresponding to a machine learning model; process, based at least on one or more language models, input data associated with the first information to generate output data representing a model card, the model card including second information describing the machine learning model; and perform, based at least on the model card, one or more operations associated with the machine learning model.

11. The system of claim 10, wherein, The first information is obtained at least by: obtaining one or more queries associated with one or more fields included in the model card; generating one or more first embeddings associated with the one or more queries; and retrieving, from the one or more databases, one or more second embeddings related to the one or more first embeddings, the one or more second embeddings being associated with the first information.

12. The system of claim 10, wherein, The one or more processors are further to: obtain a template including a format for generating the model card, wherein: the model card is further generated based at least on the one or more language models processing second input data representing the template; and the model card includes the second information arranged according to the format from the template.

13. The system of claim 10, wherein, The one or more processors are further to: obtain a second model card associated with the machine learning model, the second model card including third information describing the machine learning model, wherein: the model card is further generated based at least on the one or more language models processing second input data representing the second model card; and at least a portion of the second information included in the model card includes updated information compared to the third information included in the second model card.

14. The system of claim 10, wherein, Generating the model card includes: generating initial output data based at least on the one or more language models processing the input data; obtaining second input data representing at least one of a template associated with the model card or a second model card associated with the machine learning model; and generating, based at least on the one or more language models, the output data representing the model card from the initial output data and the second input data.

15. The system of claim 10, wherein, The one or more processors are further to: obtain third information associated with one or more second machine learning models, wherein the model card is further generated based at least on the one or more language models from second input data associated with the second information.

16. The system of claim 15, wherein, The second information is obtained at least by extracting the second information from at least one of source code associated with the one or more second machine learning models, one or more documents associated with the one or more second machine learning models, or one or more model cards associated with the one or more second machine learning models based at least on the first information.

17. The system of claim 10, wherein, Performing the one or more operations includes at least one of: storing the model card in association with the machine learning model; or determining, based at least on at least one of one or more policies or one or more capabilities associated with one or more computing devices and the model card, whether to provide the model card to the one or more computing devices.

18. The system of claim 10, wherein, The system is included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing optical transport simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using edge devices; a system implemented using robots; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more visual language models (VLMs); a system for performing operations using one or more multi-modal language models; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

19. One or more processors comprising: processing circuitry to: generate one or more embeddings associated with information describing a machine learning model; generate, based at least on one or more language models, output data representing a model card from input data associated with the one or more embeddings, the model card including at least a portion of the information describing the machine learning model; and store the model card in association with the machine learning model.

20. The one or more processors of claim 19, wherein, The one or more processors are included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing optical transport simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using edge devices; a system implemented using robots; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more visual language models (VLMs); a system for performing operations using one or more multimodal language models; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Images