Analyzing software architecture information using language models

Abstract

In various examples, analysis of software architecture information using a language model is described herein. Systems and methods are disclosed that parse architecture information associated with software, such as a software architecture document (SWAD), a design document, and / or source code, to generate a relational graph associated with the architecture information. The systems and methods may then analyze the architectural information using the relationship graph and one or more language models. For example, one or more cues associated with analysis architecture information may be obtained, where a single cue is associated with performing one or more analysis tasks. The one or more language models may then process input data representing at least a portion of the architectural information (e.g., determined using the relationship graph) and the one or more cues to determine information associated with the task.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 696,544, filed September 19, 2024, the entirety of which is hereby incorporated by reference. BACKGROUND

[0003] Software architecture documents provide information about how software is constructed, such as by describing the requirements, functions, and / or other components or features of the software. As such, software architecture documents can be used for a variety of purposes, including verifying the correctness of a software architecture and / or identifying security weaknesses associated with the software. For example, a user can review a software architecture document using a checklist that indicates different verification checks that should be performed against the software architecture and / or different weaknesses that can arise with the software in order to identify safety, security, and / or performance correctness and / or weaknesses associated with the software. However, because a software architecture document can contain hundreds and / or thousands of diagrams and / or textual descriptions, requiring a user to perform such a process requires a significant amount of time and effort and is prone to user error. SUMMARY

[0004] Embodiments of the present disclosure relate to analyzing software architecture information using language models. The present disclosure discloses a system and method that parses architecture information associated with software, such as software architecture documents (SWADs), design documents, and / or source code, to generate a relationship graph associated with the architecture information. The system and method can then analyze the architecture information using the relationship graph and one or more language models (e.g., large language models (LLMs), visual language models (VLMs), multi-modal language models (MMLMs), etc.). For example, one or more prompts associated with analyzing the architecture information can be obtained, where an individual prompt is associated with performing one or more verification tasks (e.g., checking the correctness and / or weaknesses of the software). The relationship graph can then be used to identify at least a portion of the architecture information that is relevant to the one or more prompts. Additionally, the one or more language models can process input data representing the one or more prompts, as well as the at least a portion of the architecture information, to determine information associated with the one or more tasks, such as whether the software is correct and / or includes any weaknesses.

[0005] In contrast to traditional systems, in some embodiments, the system of the present disclosure uses one or more language models to analyze architectural information, such as verifying correctness of a software architecture, identifying security vulnerabilities, and / or performing any other type of analysis. As such, in contrast to traditional systems, the system of the present disclosure can not require and / or require very little input from a user to perform the analysis, which can reduce the amount of time and / or improve accuracy. Additionally, in contrast to traditional systems, in some embodiments, the system of the present disclosure can first parse the architectural information before performing the analysis. As such, portions of the architectural information related to the analysis, such as portions describing various software requirements, can be analyzed without other portions of the architectural information that are less important to the analysis. This can again reduce the amount of time and / or improve accuracy in contrast to traditional systems. BRIEF DESCRIPTION OF DRAWINGS

[0006] The present system and method for analyzing software architectural information using language models is described in detail below with reference to the attached drawing figures, wherein:

[0007] Figure 1 An example of a process for analyzing software architectural information using one or more language models according to some embodiments of the present disclosure is shown;

[0008] Figure 2 An example of a relationship graph for generating architectural information associated with software according to some embodiments of the present disclosure is shown;

[0009] Figure 3 An example of a prompt that can be used to analyze architectural information according to some embodiments of the present disclosure is shown;

[0010] Figure 4 An example of identifying a plurality of prompts for analyzing architectural information associated with software according to embodiments of the present disclosure is shown;

[0011] Figure 5 An example of analyzing architectural information associated with software according to some embodiments of the present disclosure is shown;

[0012] Figures 6-7 A flowchart of a method for analyzing architectural information associated with software according to some embodiments of the present disclosure is shown;

[0013] Figure 8A is a block diagram of an example generative language model system suitable for implementing at least some embodiments of the present disclosure;

[0014] Figure 8B is a block diagram of an example generative language model including a transformer encoder-decoder suitable for implementing at least some embodiments of the present disclosure;

[0015] Figure 8C is a block diagram of an example generative language model that includes a decoder-only transformer architecture suitable for implementing at least some embodiments of the present disclosure;

[0016] Figure 9A is an illustration of an example autonomous vehicle in accordance with some embodiments of the present disclosure;

[0017] Figure 9B is an example of a camera position and field of view of an example autonomous vehicle in accordance with some embodiments of the present disclosure; Figure 9A

[0018] Figure 9C is an example of a camera position and field of view of an example autonomous vehicle in accordance with some embodiments of the present disclosure; Figure 9A is a block diagram of an example system architecture of an example autonomous vehicle in accordance with some embodiments of the present disclosure;

[0019] Figure 9D is a system diagram for communication between one or more cloud-based servers and an example autonomous vehicle in accordance with some embodiments of the present disclosure; Figure 9A

[0020] Figure 10 is a block diagram of an example computing device suitable for implementing at least some embodiments of the present disclosure;

[0021] Figure 11 is a block diagram of an example data center suitable for implementing at least some embodiments of the present disclosure; and

[0022] Figure 12 shows a block diagram of one or more systems that can perform at least a portion of the processes described herein in accordance with some embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] Systems and methods related to analyzing software architecture information using language models are disclosed. For example, one or more systems can obtain data representing architecture information associated with software. As described herein, the architecture information can include, but is not limited to, one or more software architecture documents (SWADs), one or more design documents, source code, and / or any other type of information associated with the software. In some examples, the one or more systems can then process at least a portion of the architecture information to generate a relationship graph associated with the software. For example, the relationship graph can include a graph, a tree, a list, a spreadsheet, and / or any other type of representation that indicates relationships between various portions of the architecture information, which can be referred to as “fragments” of the information. As described herein, the fragments can include, but are not limited to, requirements, functions, diagrams, modules, text, descriptions, and / or any other components associated with the architecture information. The one or more systems can then store the relationship graph in one or more databases. ​​

[0024] For example, if the architecture information includes one or more SWADs, one or more systems can generate tree diagrams for the different software requirements (such as security requirements, performance requirements, safety requirements, functional requirements, and / or any other type of requirement) indicated by one or more SWADs. For instance, a tree diagram can start with a requirement and then associate that requirement with relevant information such as the function associated with the requirement, the diagram associated with the requirement, the description associated with the requirement, the text associated with the requirement, and / or any other component associated with the requirement. In some examples, the tree diagram can then also associate components with one or more sub-components, such as by associating one function with another dependent function. In such examples, the tree diagram may be referred to as a fragment of architecture information and / or the individual components of the tree diagram (such as requirements, functions, etc.) may be referred to as fragments of architecture information. Additionally, one or more systems may store the tree diagrams in one or more databases.

[0025] One or more systems may also generate, retrieve, obtain, and / or receive prompts to analyze architectural information associated with the software. For example, and as described herein, prompts may be associated with performing one or more verification checks, performing one or more vulnerability checks, identifying one or more techniques for fixing any weaknesses, retrieving one or more portions of architectural information, determining whether the architectural information is complete, determining whether the architectural information is accurate, and / or performing any other analysis associated with the architectural information. Thus, a prompt may include at least an indication of one or more portions (e.g., one or more fragments) of the architectural information to be searched, one or more tasks to be performed (e.g., verifying the structure, checking for weaknesses, providing information for fixing weaknesses, etc.), one or more parameters for performing one or more tasks (e.g., instructions on how to verify the architecture, instructions on what constitutes a weakness, etc.), one or more output types for analysis, and / or any other information for analyzing the architectural information.

[0026] In some examples, one or more systems can store data representing various hints that can be used to analyze architectural information associated with software. For instance, one or more systems can store data representing a plurality of hints that are used to perform common weak point enumeration checks and / or other types of weak point checks associated with software. In these examples, one or more systems can then allow one or more users to select one or more of the hints for use in analyzing the software. For instance, one or more systems can provide information associated with the plurality of hints to one or more users, such as a list of hints and / or descriptions associated with the plurality of hints, where the one or more users can then use this information to select one or more hints for use in analyzing the architectural information. Additionally, in some examples, one or more systems can allow users to add additional information to the plurality of hints. For instance, one or more systems can allow one or more users to add information identifying one or more portions of the architectural information to search for (e.g., one or more identifiers), one or more parameters for use in performing the analysis, and / or any other relevant information.

[0027] One or more systems can then use one or more databases, one or more hints, and / or additional data (e.g., source code) to analyze the architectural information associated with the software. For instance, one or more systems can input data associated with a hint (e.g., tokens, embeddings, etc.) and at least a portion of the architectural information stored in one or more databases into one or more language models. As described herein, one or more language models can include any type of language model, such as a recurrent neural network, a retrieval-augmented generation model, a transformer model, a visual language model, a large language model, etc. One or more language models can then process the input data and generate and / or output data (e.g., tokens, embeddings, etc.) associated with text related to the hint based at least on this processing.

[0028] For a first example, if the hint is associated with determining whether there is a weak point associated with a security requirement of the software, the output can indicate whether the security requirement includes the weak point and / or one or more techniques for fixing the weak point. For a second example, if the hint is associated with determining which components can access a function, the output can indicate a list of the components. Still, for a third example, if the hint is associated with verifying whether an architecture of the software is accurate, the output can indicate whether the architecture is accurate, indicate one or more issues if the architecture is not accurate, and / or provide one or more techniques for fixing the one or more issues.

[0029] In some examples, such as to improve the performance of the analysis, one or more systems can use the relationship graph stored in one or more databases to identify one or more portions of the architectural information to process when performing the analysis. For instance, one or more systems can first input data representing the prompt into one or more language models. Then, the one or more language models can process the input data and generate, based at least on the processing, output data indicating one or more identifiers for one or more portions of the architectural information. For example, the identifiers can indicate requirements, functions, diagrams, text, descriptions, etc. from the prompt. Then, the one or more systems can use the output data and the relationship graph to identify one or more snippets associated with the prompt. Additionally, the one or more systems can then input data (such as text and / or source code) associated with the one or more identified snippets and the prompt when performing the analysis. In this way, the one or more language models process one or more portions of the architectural information that are relevant to the prompt without having to process one or more other portions of the architectural information that are not relevant and / or less relevant to the prompt, which can improve the accuracy of the one or more language models.

[0030] In some examples, these processes can continue to repeat for any number of prompts. For example, one or more systems can retrieve (such as based on one or more user inputs) a list of prompts for analyzing architectural information. Then, the one or more systems can perform the analysis using a plurality of prompts included in the list. For example, the one or more systems can analyze the architectural information using a first prompt, a second prompt that follows, a third prompt that follows, etc. By performing these processes, the one or more systems can be able to analyze the architectural information, such as to verify the architecture of software and / or determine that there are no weaknesses associated with the software, without requiring user input and / or requiring little user input. This can provide a number of improvements, such as reducing the time needed to analyze the architectural information and / or improving the accuracy of analyzing the architectural information.

[0031] As described herein, these processes can be used for various types of technical domains. For example, if a developer is creating software that is executed in a machine such as a semi-autonomous vehicle and / or an autonomous vehicle, the developer can want to analyze one or more SWADs associated with the software to ensure that the software is correct, does not include any weaknesses, and / or for any other verification reason purposes. As such, the developer can be provided with a plurality of hints associated with common weakness enumeration checks that are associated with the machine. The developer can then select one or more of the hints for analyzing the one or more SWADs and / or update one or more selected hints with additional information. Additionally, one or more language models can process input data (e.g., a relational graph) representing the one or more SWADs and the one or more selected hints and, based at least on the processing, output data representing results associated with the analysis, such as results indicating whether the software is correct and / or whether the software includes any weaknesses.

[0032] In some examples, one or more machine learning models described herein (e.g., deep neural networks, language models, LLMs, VLMs, multi-modal language models, perception models, tracking models, fusion models, transformer models, diffusion models, encoder-only models, decoder-only models, encoder-decoder models, neural rendering field (NERF) models, etc.) can be packaged as microservices, such as inference microservices (e.g., NVIDIA NIM), which can include containers (e.g., operating system (OS) level virtualization packages) that can include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model “engine.” For example, an inference microservice can include the container itself and one or more models (e.g., weights and biases). In some instances, such as where one or more machine learning models are small enough (e.g., have a small enough number of parameters), the one or more models can be included in the container itself. In other examples, such as where one or more models are large, the one or more models can be hosted / stored in the cloud (e.g., in a data center) and / or can be hosted on-premise and / or at the edge (e.g., on a local server or computing device, but outside of the container). In these embodiments, the one or more models can be accessed via one or more APIs, such as REST APIs. As such and in some embodiments, one or more machine learning models described herein can be deployed as inference microservices to accelerate the deployment of the one or more models on any cloud, data center, or edge computing system, while ensuring the security of the data. For example, an inference microservice can include one or more APIs, preconfigured containers for ease of deployment, optimized inference engines (e.g., using standardized AI model deployment building enforcement software, such as NVIDIA’s Triton Inference Server, and / or one or more APIs for high-performance deep learning inference that can include inference runtimes and model optimization that provide low latency and high throughput for production applications, such as NVIDIA’s TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). The one or more machine learning models described herein and the accelerated infrastructure can be included as part of a microservice that can be deployed using a single command and / or orchestrated and auto-scaled using a container orchestration system on the accelerated infrastructure (e.g., on a single device up to the scale of a data center).As such, an inference microservice can include one or more machine learning models (e.g., that have been optimized for high performance inference), inference runtime software that implements one or more machine learning models and provides an output / response to an input (e.g., a plurality of user queries, a plurality of prompts, etc.), and enterprise management software that provides health checks, identity, and / or other monitoring. In some embodiments, an inference microservice can include software that performs in-place replacement and / or updates to one or more machine learning models. When replaced or updated, the software performing the replacement / update can maintain user configurations of the inference runtime software and the enterprise management software.

[0033] The systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, dirigibles, watercraft, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles, drones, and / or other vehicle types. Further, the systems and methods described herein can be used for a variety of purposes, by way of example but not limitation, for machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twin, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twin, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, and / or any other suitable application.

[0034] The disclosed embodiments can be composed of a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, marine systems, smart district monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing large language models (LLMs), systems implementing one or more visual language models (VLMs), systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in data centers, systems for performing conversational AI operations, systems for performing light transport simulation, systems for performing collaborative content creation for 3D assets, systems performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0035] Reference is made toFigure 1 , Figure 1 FIG. 1 illustrates an example of a process 100 for analyzing software architecture information using one or more language models, in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be wholly omitted or consolidated. Further, many of the elements described herein are functional entities that can be implemented as discrete components or distributed across many components, combined with other components, and implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed in any suitable order, and various functions described as being performed at a certain time can be performed at other times. Some or all of the functions can be performed by a processor executing instructions stored on a memory. For example, the functions can be implemented by a processor executing instructions stored on a memory.

[0036] For example, the process 100 can include one or more parsing components 102 obtaining architecture information 104 associated with software. As described herein, the architecture information can include, but is not limited to, one or more software architecture documents (SWADs), one or more design documents, source code, and / or any other type of information associated with the software. Additionally, the one or more parsing components 102 can include and / or use one or more machine learning models, one or more neural networks, one or more modules, one or more algorithms, one or more classifiers, one or more hardware components, one or more software components, one or more computing devices, and / or any other type of processing component configured to perform one or more of the processes described herein.

[0037] The process 100 can then include the one or more parsing components 102 processing at least a portion of the architecture information 104 to generate a relationship graph 106 associated with the software. As described herein, the relationship graph 106 can include a graph, a tree, a list, a spreadsheet, and / or any other type of representation that indicates relationships between various portions of the architecture information 104, which can be referred to as “fragments” and / or “components” of the information. For example, a fragment can include, but is not limited to, a requirement, a function, a diagram, text, a module, a description, and / or any other component associated with the architecture information. The process 100 can then include storing the relationship graph 106 in one or more databases 108.

[0038] For example, Figure 2An example of a relationship diagram 202 for generating architectural information (e.g., architectural information 104) associated with software, according to some embodiments of the present disclosure, is illustrated. As shown, one or more parsing components 102 may process the architectural information to identify requirements 204 associated with the software, such as security requirements, performance requirements, safety requirements, functional requirements, and / or any other type of requirements. For a first example, one or more parsing components 102 may identify performance requirements, indicating requirements such as the amount of processing resources that can be utilized, the amount of memory resources that can be utilized, and the maximum latency for performing one or more operations. For a second example, one or more parsing components 102 may identify security requirements, indicating requirements such as how security keys should be generated, which computing devices can store security keys, and how security keys should be encrypted before being transmitted over a network. Still, for a third example, one or more parsing components 102 may identify functional requirements, indicating tasks that the software should perform and / or one or more functional requirements to be implemented to perform those tasks. While these are just a few examples of requirements associated with software, in other examples, one or more parsing components 102 can identify any other type of requirement.

[0039] One or more parsing components 102 can then generate a relationship diagram 202 to include a first node indicating requirement 204 and / or to include information describing requirement 204. One or more parsing components 102 can then further process architectural information to identify components associated with requirement 204, such as functions, diagrams, text, descriptions, and / or any other type of information from the architectural information. For example, in Figure 2 In the example, one or more parsing components 102 may determine that at least a first function 206, a second function 208, and additional information 210 are related to requirement 204. Thus, one or more parsing components 102 may generate a relationship graph 202 to include a second node associated with the first function 206, a third node associated with the second function 208, and a fourth node associated with the additional information 210. This process may then be repeated to identify secondary components associated with requirement 204. For example, one or more parsing components 102 may determine that at least a third function 212, figure 214, and additional information 216 are related to the second function 206. Thus, one or more parsing components 102 may generate a relationship graph 202 to include a fifth node associated with the third function 212, a sixth node associated with figure 214, and a seventh node associated with the additional information 216.

[0040] In some examples, the process can then continue to repeat in order to identify additional components of the architectural information related to the requirements 204 for further generating the relationship graph 202. As such, by performing such a process, the relationship graph 202 can represent at least some (e.g., all) of the architectural information required to analyze the requirements 204, such as verifying the requirements 204 and / or determining whether there are any vulnerabilities associated with the requirements 204. In some examples, one or more of the parsing components 102 can perform similar processes to generate any number of relationship graphs associated with any number of requirements, functions, graphs, and / or other components of the architectural information.

[0041] While Figure 2 Examples of the relationship graph 202 illustrate the relationship graph 202 as including a tree graph, where the nodes correspond to components and the edges indicate relationships between the components, in other examples, any other type of graph can be used to represent the architectural information associated with the requirements 204. Additionally, in some examples, the relationship graph 202 can represent a segment of the architectural information and / or each of the nodes of the relationship graph 202 can represent a segment of the architectural information.

[0042] Referring back to Figure 1 Examples of the process 100, the one or more prompting components 110 can generate one or more prompts 112 for analyzing the architectural information 104. For instance, in some embodiments, the prompts can be associated with performing one or more verification checks, performing one or more vulnerability checks, identifying one or more techniques for remediating any vulnerabilities, retrieving one or more portions of the architectural information 104, determining whether the architectural information 104 is complete, determining whether the architectural information 104 is accurate, and / or performing any other analysis associated with the architectural information 104. As such, the prompts 112 can include at least an indication of one or more portions of the architectural information 104 to search (e.g., one or more segments), one or more tasks to perform (e.g., verify structure, check for vulnerabilities, provide information for remediating vulnerabilities, etc.), one or more parameters for performing the one or more tasks (e.g., an indication of how to verify the architecture, an indication of what constitutes a vulnerability, etc.), one or more output types for the analysis, and / or any other information for analyzing the architectural information 104.

[0043] For instance, Figure 3An example of a hint 302 that can be used to analyze architectural information (e.g., architectural information 104) is illustrated in accordance with some embodiments of the present disclosure. As shown, the hint 302 can include text that describes one or more tasks 304 to be performed in connection with the analysis. For example, if the hint 302 is associated with performing a security check, the text associated with the one or more tasks 304 can include instructions to perform a security check to “analyze architectural information to identify any potential violations using one or more rules.” Additionally, the one or more tasks 304 can indicate a type of security check to be performed, such as “lack of privilege separation,” “failure to enforce secure default configuration,” “improper isolation of critical functions,” and / or any other type of security check.

[0044] The hint 302 can also include text that describes one or more parameters 306 for performing the one or more tasks 304. For a first example, if the task 304 is associated with identifying a “lack of privilege separation,” the one or more parameters 306 can indicate a technique for identifying a lack of privileges. For example, the one or more parameters 306 can indicate that the analysis for a lack of privileges is performed by “checking whether any component or module has more access rights or privileges than it needs for its function.” For a second example, if the task 304 is associated with identifying a “failure to enforce secure default configuration,” the one or more parameters 306 can indicate a technique for identifying a failure to enforce a secure default configuration. For example, the one or more parameters 306 can indicate that the analysis for a lack of secure default values is performed by “identifying whether a system or component is by default set to a non-secure setting or configuration, specifically in access control or network communication.”

[0045] The hint 302 can also include text that describes one or more identifiers 308 of portions of the architectural information to be processed when performing the analysis. As described herein, the identifiers 308 can include, but are not limited to, names, locations, addresses, numerical identifiers, alphanumeric identifiers, and / or any other type of identifier that can be used to identify a portion of the architectural information. For example, the identifiers 308 can identify requirements, functions, modules, diagrams, and / or any other component described by the architectural information. For example, an identifier 308 associated with a function can include “DOS_FUNC_GPU_OPERATION” (and / or any other type of identifier).

[0046] The prompt 302 can also include text describing one or more output types 310 generated based at least on the analysis. For the first example, if the task 304 is associated with identifying vulnerabilities associated with the software, the output types 310 can include a list of vulnerabilities and / or techniques for fixing the vulnerabilities. For the second example, if the task 304 is associated with identifying security issues using the key, the output types 310 can include information about how the key is insecure and / or techniques for better securing the key. Although Figure 3 Examples of the prompt 302 are illustrated as including one or more tasks 304, one or more parameters 306, one or more identifiers 308, and one or more outputs 310, in other examples, the prompt 302 can include additional and / or alternative information.

[0047] Referring back to the example of Figure 1 In the example of FIG. 1, the one or more prompt components 110 can use various techniques to generate the one or more prompts 112. For instance, the one or more prompt components 110 can obtain prompt data 114 representing a plurality of prompts that can be used to analyze the architectural information 104. For example, the prompt data 114 can represent at least a first prompt associated with one or more first tasks, a second prompt associated with one or more second tasks, a third prompt associated with one or more third tasks, and so on. The one or more prompt components 110 can then receive input data 116 representing one or more inputs associated with selecting one or more of the prompts. In some examples, the one or more inputs can be associated with selecting one or more actual prompts. Additionally or alternatively, in some examples, the one or more inputs can indicate one or more tasks to perform with respect to the architectural information 104. In these examples, the one or more prompt components 110 can then select one or more prompts 112 associated with performing the one or more tasks.

[0048] For further details, Figure 4An example of identifying multiple prompts for analyzing software-related architectural information according to embodiments of the present disclosure is illustrated. As shown, one or more prompt components 110 may obtain prompt data 402 (which may include and / or be similar to prompt data 114) representing multiple prompts 404(1)-(N) (also referred to in the singular as "prompt 404" or in the plural as "multiple prompts 404") and / or descriptions 406(1)-(N) (also referred to in the singular as "description 406" or in the plural as "multiple descriptions 406") respectively associated with prompts 404. In some examples, the multiple descriptions 406 may provide details about the multiple prompts 404, such as the task being performed, parameters associated with the task, and / or any other details associated with the multiple prompts 404. One or more users may then select at least a portion of the multiple prompts 404 for analyzing software-related architectural information.

[0049] For the first example, the user device may display content representing at least a portion of multiple prompts 404 and / or at least a portion of multiple descriptions 406. One or more users can then use the multiple prompts 404 and / or multiple descriptions 406 to select one or more prompts 404 for performing analysis. For the second example, one or more users can provide details about the type of analysis to be performed with respect to architectural information, such as different tasks. One or more prompt components 110 can then use details about the analysis type, the multiple prompts 404 and / or multiple descriptions 406 to again select one or more prompts 404 associated with the type of analysis to be performed. While these are just a few example techniques for how one or more prompt components 110 select multiple prompts 404, in other examples, one or more prompt components 110 may use additional and / or alternative techniques to select multiple prompts 404.

[0050] Return to reference Figure 1 For example, in some examples, one or more prompt components 110 may allow one or more users to customize one or more prompts 112. For the first example and as described herein, prompt 112 may include indications of one or more portions of the architectural information 104 to be searched when performing analysis (e.g., from...). Figure 3one or more identifiers 308 of the prompt 302 of the first example). As such, the one or more prompting components 110 can receive input data 116 representing one or more identifiers associated with one or more portions of the architectural information 104. Additionally, the one or more prompting components 110 can input the one or more identifiers into the prompt 112. For the second example, the prompt 112 can include one or more parameters 306 of the prompt 302 of the first example. As such, the one or more prompting components 110 can receive input data 116 representing one or more of the parameters. Additionally, the one or more prompting components 110 can input the one or more parameters into the prompt 112. These are just a few example techniques for how the one or more prompting components 110 can customize the one or more prompts 112, in other examples, the one or more prompting components 110 can also use additional and / or alternative techniques to customize the one or more prompts 112. Figure 3

[0051] The process 100 can include analyzing the architectural information 104 using one or more analysis components 118 using at least the one or more prompts 112. As described herein, in some examples, the one or more analysis components 118 can analyze the architectural information 104 using one or more language models 120. Additionally, the one or more language models 120 can include any type of language model, such as a retrieval-augmented generation model, a transformer model, a visual language model, a large language model, etc. In some examples, the one or more language models 120 can include one or more general purpose language models that are not specifically trained to analyze the architectural information 104. However, in other examples, the one or more language models 120 can be trained to analyze the architectural information 104. For instance, the one or more language models 120 can be trained to perform various tasks associated with the analysis, such as tasks associated with verifying the correctness of an architecture, tasks associated with identifying weaknesses in software, etc.

[0052] ​To analyze the architectural information 104, the one or more analysis components 118 can input data associated with the prompt 112 (e.g., tokens, embeddings, etc.) and at least a portion of the architectural information 104 stored in the one or more databases 108 into the one or more language models 120. The one or more language models 120 can then process the input data and generate and / or output, based at least on the processing, analysis data 122 (e.g., tokens, embeddings, content, etc.) associated with the text related to the prompt 112. For the first example, if the prompt 112 is associated with determining whether there is a vulnerability associated with a security requirement of the software, the output can indicate whether the security requirement includes the vulnerability and / or one or more techniques for fixing the vulnerability. For the second example, if the prompt 112 is associated with determining which components can access a function, the output can indicate a list of the components. Still, for the third example, if the prompt is associated with verifying whether an architecture of the software is accurate, the output can indicate whether the architecture is accurate and / or one or more issues if the architecture is not accurate.

[0053] In some examples, such as to improve performance of the analysis, the one or more analysis components 118 can use the relationship graph 106 stored in the one or more databases 108 to identify one or more portions of the architectural information 104 to process when performing the analysis. For instance, the one or more analysis components 118 can first input data representing the prompt 112 into the one or more language models 120. The one or more language models 120 can then process the input data and generate, based at least on the processing, output data indicating one or more identifiers for one or more portions of the architectural information 104. For example, the identifiers can indicate a requirement, a function, a graph, text, a description, etc. from the prompt 112. The one or more analysis components 118 can then use the output data and the relationship graph 106 to identify one or more segments associated with the prompt 112. For example, the one or more analysis components 118 can match the one or more identifiers to one or more segments. The one or more analysis components 118 can then identify one or more of the segments and / or one or more related segments based on relationships from the relationship graph 106. Additionally, the one or more analysis components 118 can then input data associated with the one or more identified segments, such as text and / or source code, when performing the analysis.

[0054] For further details, Figure 5 FIGURE 1 illustrates an example of one or more analysis components 118 analyzing architectural information associated with software according to some embodiments of the present disclosure. In some examples, Figure 5Examples can be associated with an LLM agent that performs iterative decision making to perform one or more of the processes described herein.

[0055] As illustrated, one or more analysis components 118 can receive a prompt 502 selected by one or more users 504 (which can include and / or be similar to one or more prompts 112). The one or more analysis components 118 can then determine how to proceed with the analysis, which is represented by a thought 506. For example, the one or more analysis components 118 can determine to process the prompt 502 using one or more language models 120. Next, the one or more analysis components 118 can perform an action 508 associated with the decision of how to proceed. For example, the one or more analysis components 118 can input data representing the prompt 502 into the one or more language models 120. The one or more language models 120 can then process the input data and generate output data associated with the prompt 502. Next, the one or more analysis components 118 can observe 510 the results of the action 508. For example, the one or more analysis components 118 can determine that the output data represents one or more identifiers of one or more portions of architectural information to analyze.

[0056] The process can then continue to repeat as the one or more analysis components 118 continue to process the prompt 502. For instance, the one or more analysis components 118 can again determine how to proceed using the results observed from the initial action, which is again represented by a thought 506. For example, the one or more analysis components 118 can determine to use the one or more identifiers to retrieve the one or more portions of architectural information. The one or more analysis components 118 can then perform another action 508. For example, the one or more analysis components 118 can input data representing the prompt 502 and the one or more portions of architectural information into the one or more language models 120. The one or more language models 120 can then process the input data and generate output data associated with the prompt. Next, the one or more analysis components 118 can observe 510 the results of the additional action 508. For example, the one or more analysis components 118 can determine that the output data represents a result of the analysis. As such, the one or more analysis components 118 can provide the result to the one or more users 504, as indicated by an analysis 512.

[0057] While this example describes executing the process twice to retrieve one or more results for the user, in other examples, the process may be executed any number of times to retrieve one or more results for the user. For example, the process may be repeated multiple times so that one or more analysis components 118 retrieve one or more of the most relevant parts of the architectural information for processing by one or more language models 120.

[0058] Return to reference Figure 1 While the examples above describe using a relationship graph 106 to identify one or more portions of the architecture information 104 to be processed, other techniques may be used in other examples to identify one or more portions of the architecture information 104. In a first example, one or more parsing components 102 may segment the architecture information 104 into text chunks, such as words, sentences, paragraphs, pages, and / or any other text blocks. One or more analysis components 118 may then input the text chunks into one or more language models 120 when processing is performed. In a second example, one or more parsing components 102 may generate embeddings representing text from the architecture information 104. Additionally, one or more analysis components 118 may generate additional embeddings associated with multiple cues 112. One or more analysis components 118 may then use the embeddings to identify one or more portions of the architecture information 104 for processing, such as based on the similarity between embeddings.

[0059] In some examples, one or more analysis components 118 may then proceed to perform these processes on one or more additional prompts 112. For example, if one or more prompts 112 are included in a list, one or more analysis components 118 may perform similar processes on one or more (e.g., each) of the one or more prompts 112 included in the list. In this way, one or more analysis components 118 may perform a detailed analysis of the architectural information 104, such as based on the needs of one or more users.

[0060] Then, process 100 may include providing analytical data 122 to one or more user devices 124. In this way, one or more user devices 124 may present results associated with the analysis to one or more users, who can view the results, verify the results, and / or determine how to use the results to improve architectural information 104 and / or software. For a first example, if the results indicate the existence of weaknesses associated with the software and / or one or more techniques for fixing those weaknesses, one or more users may use the results to fix the software, such as according to one or more techniques. For a second example, if the results indicate that a portion of the architecture is inaccurate, one or more users may use one or more techniques to use the results to correct the architecture.

[0061] Now, referring to Figure 6 and Figure 7 Each block of the methods 600 and 700 described herein comprises a computational process that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The methods 600 and 700 can also be embodied as computer-usable instructions stored on computer storage media. The methods 600 and 700 can be provided by a standalone application, a service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Figure 1 The methods 600 and 700 are described with respect to the system of

[0062] Figure 6 A flow diagram of a method 600 for analyzing architectural information associated with software, in accordance with some embodiments of the present disclosure, is illustrated. At block B602, the method 600 can include obtaining architectural information associated with software and a hint associated with performing one or more tasks with respect to the architectural information. For instance, the one or more analysis components 118 can receive the hint 112 and the architectural information 104. As described herein, in some examples, the one or more analysis components 118 can receive the architectural information 104 parsed into a relational graph 106 stored in the one or more databases 108.

[0063] At block B604, the method 600 can include generating one or more input tokens associated with at least a portion of the architectural information 104 and the hint. For instance, the one or more analysis components 118 can generate one or more input tokens associated with at least a portion of the architectural information 104 and the hint 112. In some examples, the one or more analysis components 118 can first use the relational graph 106 to identify at least a portion of the architectural information 104. For example, the one or more analysis components 118 can determine that one or more identifiers from the hint 112 correspond to one or more segments from the relational graph 106. The one or more analysis components 118 can then retrieve one or more portions of the architectural information 104 associated with the one or more segments, source code associated with the one or more portions of the architectural information 104, and / or any other data related to the one or more segments.

[0064] At block B606, the method 600 can include generating, using one or more language models and based at least on the one or more input tokens, one or more output tokens representing information associated with the one or more tasks. For instance, the one or more analysis components 118 can input the one or more tokens into the one or more language models 120. The one or more language models 120 can then process the one or more tokens and generate one or more output tokens associated with information. Additionally, the one or more language models 120, the one or more analysis components 118, and / or another component can process the one or more output tokens in order to determine information associated with the one or more tasks.

[0065] At block B608, the method 600 can include performing one or more operations based at least on the information. For instance, the one or more analysis components 118 can generate and / or output analysis data 122 representing at least the information. The information can then be provided to one or more users, used to update architectural information 104, and / or used to perform any other type of operation.

[0066] Figure 7 FIG. illustrates a flowchart of a method 700 for analyzing architectural information associated with software, in accordance with some embodiments of the present disclosure. At block B702, the method 700 can include obtaining architectural information associated with software. For instance, the one or more analysis components 118 can receive architectural information 104 associated with software. As described herein, the architectural information can include, but is not limited to, one or more software architecture documents, one or more design documents, source code, and / or any other type of information associated with software. Additionally, in some examples, the one or more analysis components 118 can receive architectural information 104 that is parsed into a relational graph 106 stored in one or more databases 108.

[0067] At block B704, the method 700 can include determining, using one or more language models and based at least on a prompt associated with performing one or more tasks corresponding to the architectural information, information associated with the one or more tasks. For instance, the one or more analysis components 118 can input data associated with the architectural information and the prompt into the one or more language models 120. The one or more language models 120 can then process the input data and generate output data representing information.

[0068] At block B706, the method 700 can include performing one or more operations based at least on the information. For instance, one or more analysis components 118 can generate and / or output analysis data 122 that represents at least the information. The information can then be provided to one or more users, used to update architecture information 104, and / or used to perform any other type of operation.

[0069] Example Language Models

[0070] In at least some embodiments, language models can be implemented, such as large language models (LLMs), visual language models (VLMs), multi-modal language models (MMLMs), and / or other types of generative artificial intelligence (AI). These models can be capable of understanding, summarizing, translating, and / or otherwise generating text (e.g., natural language text, code, etc.), images, videos, computer-aided design (CAD) assets, OMNIVERSE and / or METAVERSE file information (e.g., USD format, such as OpenUSD), and / or the like based on context provided in an input prompt or query. In embodiments, these language models can be considered “large” in that these models are trained on massive datasets and have architectures with a large number of learnable network parameters (weights and biases) - e.g., millions or billions of parameters. LLMs / VLMs / MMLMs / etc. can be implemented for summarizing textual data, analyzing data (e.g., text, images, videos, etc.), and extracting insights from data (e.g., text, images, videos, etc.), as well as generating new text / images / videos / etc. in a user-specified style, tone, and / or format. In embodiments, LLMs / VLMs / MMLMs / etc. of the present disclosure can be specialized for text processing, while in other embodiments, multi-modal LLMs can be implemented to accept, understand, and / or generate text and / or other types of content, such as images, audio, 2D and / or 3D data (e.g., USD format), and / or videos. For instance, a visual language model (VLM) or more specifically a multi-modal language model (MMLM) can be implemented to accept image, video, audio, text, 3D designs (e.g., CAD), and / or other input data types and / or generate or output image, video, audio, text, 3D designs, and / or other output data types.

[0071] Various types of LLM / VLM / MMLM / etc. architectures can be implemented in various embodiments. For example, different architectures can be implemented that use different techniques to understand and generate output (e.g., text, audio, video, images, 2D and / or 3D design or asset data, etc.). In some embodiments, LLM / VLM / MMLM / etc. architectures (e.g., recurrent neural networks (RNNs) or long short-term memory networks (LSTMs)) can be used, while in other embodiments, transformer architectures (e.g., architectures that rely on self-attention and / or cross-attention (e.g., between contextual data and textual data) mechanisms) can be used to understand and recognize relationships between words or tokens and / or contextual data (e.g., other text, video, images, design data, USD, etc.). One or more generative processing pipelines that include LLM / VLM / MMLM / etc. can also include one or more diffusion blocks (e.g., denoisers). LLM / VLM / MMLM / etc. of the present disclosure can include encoder and / or decoder blocks. For example, discriminative or encoder-only models (e.g., BERT (Bidirectional Encoder Representations from Transformers)) can be implemented for tasks involving language understanding (e.g., classification, sentiment analysis, question answering, and named entity recognition). As another example, generative or decoder-only models (e.g., GPT (Generative Pretrained Transformer)) can be implemented for tasks involving language and content generation (e.g., text completion, story generation, and dialogue generation). LLM / VLM / MMLM / etc. that include both encoder and decoder components (e.g., T5 (Text-to-Text Transformer)) can be implemented to understand and generate content, such as for translation and summarization. These examples are not intended to be limiting, and any architecture type (including, but not limited to, the architecture types described herein) can be implemented depending on the particular embodiment and the tasks being performed using the LLM / VLM / MMLM / etc.

[0072] In various embodiments, LLMs / VLMs / MMLMs / etc. can be trained using unsupervised learning, where the LLMs / VLMs / MMLMs / etc. learn patterns from large amounts of unlabeled text / audio / video / image / design / USD / etc. data. As a result of extensive training, in embodiments, the models can not require task-specific or domain-specific training. LLMs / VLMs / MMLMs / etc. that are extensively pre-trained on large amounts of unlabeled data can be referred to as base models, and can be good at a variety of tasks, such as question answering, summarization, filling in missing information, translation, image / video / design / USD / data generation. Some LLMs / VLMs / MMLMs / etc. can be customized for specific use cases using techniques such as prompt tuning, fine-tuning, retrieval-augmented generation (RAG), adding adapters (e.g., custom neural networks and / or neural network layers for tuning or adjusting prompts or tokens to bias the language model toward a particular task or domain), and / or using optimization models for specific tasks and / or for other fine-tuning or customization techniques within a particular domain.

[0073] In some embodiments, LLMs / VLMs / MMLMs / etc. of the present disclosure can be implemented using various model alignment techniques. For example, in some embodiments, guardrails can be implemented to identify incorrect or unwanted inputs (e.g., prompts) and / or outputs of the models. In this process, the system can use guardrails and / or other model alignment techniques to prevent particular unwanted inputs from being processed using the LLMs / VLMs / MMLMs / etc., and / or to prevent outputs or presentations (e.g., displays, audio outputs, etc.) of information generated using the LLMs / VLMs / MMLMs / etc. In some embodiments, one or more additional models (or layers thereof) can be implemented to identify problems with inputs and / or outputs of the models. For example, these “guard” models can be trained to identify “safe” or otherwise ok or wanted inputs and / or outputs and / or “unsafe” or otherwise unwanted inputs and / or outputs for a particular application / implementation. Thus, LLMs / VLMs / MMLMs / etc. of the present disclosure can be less likely to output language / text / audio / video / design data / USD data / etc. that can be offensive, vulgar, inappropriate, unsafe, out of scope, and / or otherwise unwanted for a particular application / implementation.

[0074] In some embodiments, the LLM / VLM / etc. can be configured to or have access to or use one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc. For example, for certain tasks or operations for which the model is not ideally suited, the model can have instructions for accessing one or more plugins (e.g., third-party plugins) to assist in processing the current input (e.g., as a result of training, and / or based on instructions in a given prompt). In such examples, when at least a portion of the prompt is related to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs) to retrieve relevant information. Another example is if at least a portion of the response requires mathematical calculations, the model can access one or more mathematical plugins or APIs to assist in solving the problem, which can then be used in the output of the model from the response of the plugin and / or API. This process can repeat (e.g., recursively) for any number of iterations and using any number of plugins and / or APIs until a response to the input prompt can be generated that addresses each inquiry / question / request / process / operation / etc. Thus, the model can rely not only on its own knowledge gained from training on large datasets, but also on the expertise or optimized properties of one or more external resources (e.g., APIs, plugins, etc.).

[0075] In some embodiments, multiple language models (e.g., LLMs / VLMs / MMLMs / etc., multiple instances of the same language model, and / or multiple prompts provided to the same language model or instance of the same language model) can be implemented, executed, or accessed (e.g., using one or more plugins, user interfaces, APIs, databases, data stores, repositories, etc.) to provide outputs responsive to the same query or responsive to separate portions of the query. In at least one embodiment, the same input query and prompts (e.g., a set of constraints, condition generators, etc.) can be provided to multiple language models (e.g., language models with different architectures, language models trained on different (e.g., updated) data corpora). In one or more embodiments, the language models can be different versions of the same base model. In one or more embodiments, at least one language model can be instantiated as multiple agents, e.g., more than one prompt can be provided to constrain, guide, or otherwise influence the style, content, or character of the output provided, etc. In one or more example non-limiting embodiments, the same language model can be required to provide outputs corresponding to different roles, perspectives, characters, or having different knowledge bases, etc. as defined by the prompts provided.

[0076] In any of such embodiments, the outputs of two or more (e.g., each) language models, two or more versions of at least one language model, two or more instantiations of at least one language model, and / or two or more prompts provided to at least one language model can be further processed, e.g., aggregated, compared, or filtered, or used to determine (and provide) a consensus response. In one or more embodiments, the output from one language model (or version, instantiation, or agent) can be provided as input to another language model for further processing and / or validation. In one or more embodiments, a language model can be required to generate or otherwise obtain an output regarding an input source material, where the output is associated with the input source material. Such association can include, for example, generating an embedding (e.g., as metadata) a caption or text portion within an input source text or image. In one or more embodiments, the output of a language model can be used to determine the validity of an input source material for further processing or inclusion in a dataset. For example, a language model can be used to assess the presence (or absence) of a target word in a text portion or the presence (or absence) of an object in an image, where the text or image is annotated to indicate such presence (or lack thereof). Alternatively, a determination from a language model can be used to determine whether a source material should be included in a curated dataset, for example, but not limited to.

[0077] Figure 8A is a block diagram of an example generative language model system 800 suitable for implementing at least some embodiments of the present disclosure. In Figure 8A In the example shown, the generative language model system 800 includes a retrieval-augmented generation (RAG) component 892, an input processor 805, a tokenizer 810, an embedding component 820, a plug-in / API 895, and a generative language model (LM) 830 (which can include a LLM, a VLM, a multi-modal LM, etc.).

[0078] At a high level, input processor 805 can receive input 801 that includes text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasound, etc.), 3D design data, CAD data, Universal Scene Description (USD) data (e.g., OpenUSD, etc.), depending on the architecture of generative LM 830 (e.g., LLM / VLM / MMLM / etc.). In some embodiments, input 801 includes plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, input 801 can include numerical sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., table format, JSON, or XML). In some implementations where generative LM 830 is capable of processing multi-modal input, input 801 can combine text (or can omit text) with image data, audio data, video data, design data, USD data, and / or other types of input data such as, but not limited to, the data described herein. Taking the example of raw input text, input processor 805 can prepare the raw input text in various ways. For example, input processor 805 can perform various types of text filtering to remove noise (e.g., special characters, punctuation, HTML tags, stop words, portions of images, portions of audio, etc.) from the relevant textual content. In examples involving stop words (commonly used words that tend to have little semantic meaning), input processor 805 can remove stop words to reduce noise and cause generative LM 830 to focus on more meaningful content. Input processor 805 can apply text normalization, e.g., by converting all characters to lowercase, removing diacritics, and / or handling special cases (such as abbreviations or contractions) to ensure consistency. These are just a few examples, and other types of input processing can be applied.

[0079] In some embodiments, RAG component 892 (which can include one or more RAG models, and / or which can use generative LM 830 itself to perform) can be used to retrieve additional information to be used as part of input 801 or a prompt. The RAG can be used to augment the input to the LLM / VLM / MMLM / etc. with external knowledge in order to make the answer to a particular question or query or request more relevant, e.g., in cases where specific knowledge is needed. RAG component 892 can obtain this additional information (e.g., underlying information such as underlying text / image / video / audio / USD / CAD / etc.) from one or more external sources, which can then be fed to the LLM / VLM / MMLM / etc. along with the prompt to improve the accuracy of the model’s response or output.

[0080] For example, in some embodiments, input 801 can be generated using a query or model input (e.g., a question, a request, etc.) in addition to data retrieved using RAG component 892. In some embodiments, input processor 805 can analyze input 801 and communicate with RAG component 892 (or in embodiments, RAG component 892 can be part of input processor 805) in order to identify relevant text and / or other data to provide to generative LM 830 as additional context or source of information, typically from which to identify a response, answer, or output 890. For example, when input indicates that a user is interested in the required tire pressure for a particular make and model of vehicle, RAG component 892 can use a RAG model to perform a vector search, e.g., in an embedding space, to retrieve tire pressure information or text corresponding thereto from a digital (embedded) version of the user manual for that particular vehicle make and model. Similarly, when a user revisits a chatbot related to a particular product sale or service, RAG component 892 can retrieve a previously stored dialog history (or at least an abridged version thereof) and provide the previous dialog history along with the current inquiry / request as part of input 801 to generative LM 830.

[0081] RAG component 892 can use various RAG techniques. For example, a naive RAG (RAG) can be used in which documents are indexed, chunked, and applied to an embedding model to generate embeddings corresponding to the chunks. A user query can also be applied to the embedding model and / or another embedding model of RAG component 892, and the embeddings of the chunks can be compared to the embedding of the query to identify the most similar / most relevant embeddings to the query, which can be provided to generative LM 830 to generate an output.

[0082] In some embodiments, more advanced RAG techniques can be used. For example, chunks can undergo pre-retrieval processes (e.g., routing, rewriting, metadata analysis, expansion, etc.) before being passed to an embedding model. Furthermore, outputs of the embedding model can be subjected to post-retrieval processes (e.g., re-ranking, prompt compression, etc.) before the final embeddings are used as comparisons to an input query.

[0083] As a further example, modular RAG techniques can be used, e.g., techniques similar to naive RAG and / or advanced RAG but also including features such as hybrid search, recursive retrieval and query engine, StepBack method, subquery and hypothetical document embeddings, etc.

[0084] As another example, a Graph RAG can use a knowledge graph as a source of contextual or factual information. The Graph RAG can be implemented using a graph database as a source of contextual information sent to the LLM / VLM / MMLM / etc. Rather than (or in addition to) providing the model with data chunks extracted from larger documents (which can result in a lack of context, factual accuracy, language accuracy, etc.), the Graph RAG can also provide structured entity information to the LLM / VLM / MMLM / etc. by combining structured entity text descriptions with their many attributes and relationships, allowing the model to gain deeper insights. In implementing a Graph RAG, the systems and methods described herein use graphs as content stores and extract relevant document chunks and ask the LLM / VLM / MMLM / etc. to use them to answer. In such embodiments, the knowledge graph can contain relevant textual content and metadata about the knowledge graph, and can also be integrated with a vector database. In some embodiments, the Graph RAG can use graphs as subject matter experts, where descriptions of concepts and entities relevant to the query / prompt can be extracted and passed to the model as semantic context. These descriptions can include relationships between concepts. In other examples, the graph can be used as a database, where a portion of the query / prompt can be mapped to a graph query, the graph query can be executed, and the LLM / VLM / MMLM / etc. can summarize the results. In such examples, the graph can store relevant factual information, and can use querying (natural language querying) and entity linking to a graph querying tool (NL-to-graph querying tool). In some embodiments, the Graph RAG (e.g., using a graph database) can be combined with standard (e.g., vector database) RAGs and / or other RAG types to benefit from multiple approaches.

[0085] In any embodiment, the RAG component 892 can implement plugins, APIs, user interfaces, and / or other functionality to perform RAGs. For example, the LLM / VLM / MMLM / etc. can use a Graph RAG plugin to run queries against a knowledge graph to extract relevant information to feed into the model, and can use a standard or vector RAG plugin to run queries against a vector database. For example, the graph database can interact with a REST interface of the plugin, such that the graph database can be decoupled from the vector database and / or embedding model.

[0086] The tokenizer 810 can segment (e.g., processed) textual data into smaller units (tokens) for subsequent analysis and processing. Depending on the implementation, a token can represent an individual word, subword, character, portion of audio / video / image / etc. Word-based tokenization divides text into individual words, treating each word as a separate token. Subword tokenization breaks down words into smaller meaningful units (e.g., prefixes, suffixes, stems), enabling the generative LM 830 to understand morphological variations and more effectively handle out-of-vocabulary words. Character-based tokenization represents each character as a separate token, enabling the generative LM 830 to process text at a fine-grained level. The choice of tokenization strategy can depend on factors such as the language being processed, the task at hand, and / or characteristics of the training dataset. Accordingly, the tokenizer 810 can transform (e.g., processed) text into a structured format according to a tokenization scheme implemented in a particular embodiment.

[0087] The embedding component 820 can transform discrete tokens into a semantic (e.g., dense, continuous vector) representation using any known embedding technique. For example, the embedding component 820 can use pre-trained word embeddings (e.g., Word2Vec, GloVe, or FastText), one-hot encoding, Term Frequency-Inverse Document Frequency (TF-IDF) encoding, one or more embedding layers of a neural network, and / or others.

[0088] In some implementations in which input 801 includes image data / video data / etc., input processor 801 can resize data to a standard size compatible with the format of the respective input channel and / or can normalize pixel values to a common range (e.g., 0 to 1) to ensure consistent representation, and embedding component 820 can encode image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations in which input 801 includes audio data, input processor 801 can resample audio files to a consistent sampling rate for uniform processing, and embedding component 820 can extract and encode audio features using any known technique, e.g., in the form of a spectrogram (e.g., a mel-spectrogram). In some implementations in which input 801 includes video data, input processor 801 can extract frames or apply resizing to extracted frames, and embedding component 820 can extract features such as optical flow embeddings or video embeddings and / or can encode temporal information or sequences of frames. In some implementations in which input 801 includes multi-modal data, embedding component 820 can fuse representations of different types of data (e.g., text, image, audio, USD, video, design, etc.) using techniques such as early fusion (concatenation), late fusion (sequential processing), attention-based fusion (e.g., self-attention, cross-attention), etc.

[0089] The generative LM 830 and / or other components of the generative LM system 800 can use different types of neural network architectures depending on the implementation. For example, a transformer-based architecture (such as used in GPT, etc. models) can be implemented, and it can include a self-attention mechanism that weights the importance of different words or tokens in the input sequence and / or a feed-forward network that processes the output of the self-attention layers, which applies a non-linear transformation to the input representation and extracts higher-level features. Some non-limiting example architectures include transformers (e.g., encoder-decoder, decoder-only, multi-modal), RNNs, LSTMs, fusion models, diffusion models, cross-modal embedding models that learn a joint embedding space, graph neural networks (GNNs), hybrid architectures that combine different types of architectures adversarial networks such as generative adversarial networks or GANs or adversarial autoencoders (AAEs) for joint distribution learning, etc. Thus, depending on the implementation and architecture, the embedding component 820 can apply the encoded representation of the input 801 to the generative LM 830, and the generative LM 830 can process the encoded representation of the input 801 to generate an output 890, which can include response text and / or other types of data.

[0090] As described herein, in some embodiments, generative LM 830 can be configured to access or use (or be able to access or use) plugins / APIs 895 (which can include one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc.). For example, for certain tasks or operations that are not ideally suited for generative LM 830, the model can have instructions (e.g., as a result of training, and / or based on instructions in a given prompt, such as instructions retrieved using RAG component 892) for accessing one or more plugins / APIs 895 (e.g., third-party plugins) to assist in processing the current input. In such examples, when at least a portion of the prompt is related to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs), send at least a portion of the prompt related to the particular plugin / API 895 to the plugin / API 895, the plugin / API 895 can process that information and return an answer to generative LM 830, which can use that response to generate output 890. This process can repeat (e.g., recursively) for any number of iterations and with any number of plugins / APIs 895 until output 890 can be generated that addresses each query / question / request / process / operation / etc. from input 801. Thus, the model can rely not only on its own knowledge obtained from training on large datasets and / or from data retrieved using RAG component 892, but also on the specialized knowledge or optimized properties of one or more external resources (e.g., plugins / APIs 895).

[0091] Figure 8B is a block diagram of an example implementation in which generative LM 830 includes a transducer encoder-decoder. For example, assume that input text (e.g., “Who discovered gravity”) is tokenized (e.g., by tokenizer 810 of Figure 8A Figure 8A

[0092] ​​In example implementations, the encoder 835 forms an encoder stack in which each encoder includes a self-attention layer and a feedforward network. In an example transducer architecture, each token (e.g., word) flows through a separate path. Thus, each encoder can accept a sequence of vectors, pass each vector through a self-attention layer, then through a feedforward network, and then up to the next encoder in the stack. Any known self-attention technique can be used. For example, to compute a self-attention score for each token (word), a query vector, a key vector, and a value vector can be created for each token, a self-attention score for a pair of tokens can be computed by taking the dot product of the query vector and the corresponding key vector, normalizing the resulting score, multiplying by the corresponding value vector, and summing the weighted value vectors. The encoder can apply multi-head attention in which the attention mechanism is applied multiple times in parallel with different learned weight matrices. Any number of encoders can be concatenated to generate a context vector that encodes the input. The attention projection layer 840 can transform the context vector into an attention vector (keys and values) for the decoder 845.

[0093] In example implementations, the decoder 845 forms a decoder stack in which each decoder includes a self-attention layer, an encoder-decoder self-attention layer that attends to relevant portions of the input sequence using the attention vectors (keys and values) from the encoder, and a feedforward network. As with the encoder 835, in an example transducer architecture, each token (e.g., word) flows through a separate path in the decoder 845. During a first pass, the decoder 845, the classifier 850, and the generation mechanism 855 can generate a first token, and the generation mechanism 855 can apply the generated token as input during a second pass. This process can be repeated iteratively, generating tokens (e.g., words) one at a time and adding them to the output of the previous pass, and applying the token embedding of the composite sequence with positional encodings as input to the decoder 845 in subsequent passes, iteratively generating one token at a time (known as autoregressive) until a symbol or token is predicted that indicates the end of the response. In each decoder, the self-attention layer is typically restricted to attending only to previous positions in the output sequence by applying a masking technique (e.g., setting future positions to negative infinity) prior to the softmax operation. In example implementations, the operation of the encoder-decoder attention layer is similar to that of the (e.g., multi-head) self-attention in the encoder 835, except that it creates its queries from the layer below it and takes its keys and values (e.g., matrices) from the output of the encoder 835.

[0094] Accordingly, the decoder 845 can output some decoded (e.g., vector) representation of the input applied during a particular pass. The classifier 850 can include a multi-class classifier that includes one or more neural network layers that project the decoded (e.g., vector) representation to respective dimensions (e.g., one dimension for each supported word or token in the output vocabulary) and a softmax operation that converts logits to probabilities. Accordingly, the generation mechanism 855 can select or sample a word or token based on the respective predicted probabilities (e.g., select the word with the highest predicted probability) and append it to the output of the previous pass, generating each word or token in sequence. The generation mechanism 855 can repeat this process, triggering successive decoder inputs and respective predictions, until a symbol or token is selected or sampled that indicates the end of the response, at which point the generation mechanism 855 can output the generated response.

[0095] Figure 8C is a block diagram of an example implementation in which the generative LM 830 includes a decoder-only transducer architecture. For example, Figure 8C The decoders 860 of Figure 8B may operate similarly to the decoders 845 of Figure 8C Each decoder 860 of Figure 8B may form a decoder stack in which each decoder includes a self-attention layer and a feedforward network. Further, a symbol or token representing the end of the input sequence (or the beginning of the output sequence) can be appended to the input sequence instead of encoding the input sequence, and the resulting sequence (e.g., respective embeddings with position encodings) can be applied to the decoders 860. Like the decoders 845 of Figure 8B Each token (e.g., word) can flow through a separate path in the decoders 860, and the decoders 860, classifier 865, and generation mechanism 870 can use autoregression to generate one token at a time in sequence until a symbol or token is predicted that represents the end of the response. The classifier 865 and generation mechanism 870 can operate similarly to the classifier 850 and generation mechanism 855 of

[0096] Example autonomous vehicle

[0097] Figure 9Ais a diagram of an example autonomous vehicle 900 according to some embodiments of the present disclosure. The autonomous vehicle 900 (alternatively referred to herein as “vehicle 900”) can include, but is not limited to, a passenger vehicle such as a car, truck, bus, first responder vehicle, shuttle, electric or motorized bicycle, motorcycle, fire vehicle, police vehicle, ambulance, boat, construction vehicle, underwater vehicle, robotic vehicle, drone, airplane, vehicle coupled to one or more trailers (e.g., a semi-truck trailer for hauling cargo), and / or another type of vehicle (e.g., a self-driving and / or passenger-holding vehicle). Autonomous vehicles are generally described in terms of levels of automation as defined by a department of the United States Department of Transportation, the National Highway Traffic Safety Administration (NHTSA), and the Society of Automotive Engineers (SAE) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806 published June 15, 2018, Standard No. J3016-201609 published September 30, 2016, and prior and future versions of this standard). The vehicle 900 can be capable of implementing functionality that complies with one or more of Levels 3-5 of autonomous driving. The vehicle 900 can be capable of implementing one or more of Levels 1-5 of autonomous driving. For example, depending on the embodiment, the vehicle 900 can be capable of implementing driver-assisted (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). The term “autonomous” as used herein can include any and / or all types of autonomy of the vehicle 900 or other machine, such as fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, providing assisted autonomy, semi-autonomous, primarily autonomous, or other designation.

[0098] The vehicle 900 can include components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of the vehicle. The vehicle 900 can include a propulsion system 950 such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. The propulsion system 950 can be connected to a drivetrain of the vehicle 900 that can include a transmission to allow propulsion of the vehicle 900. The propulsion system 950 can be controlled in response to receiving a signal from a throttle / accelerator 952.

[0099] A steering system 954, which can include a steering wheel, can be used to steer the vehicle 900 (e.g., along a desired path or route) while the propulsion system 950 is operating (e.g., while the vehicle is in motion). The steering system 954 can receive a signal from a steering actuator 956. For full automation (level 5) functionality, the steering wheel can be optional.

[0100] A braking sensor system 946 can be used to operate the vehicle brakes in response to receiving a signal from a braking actuator 948 and / or a braking sensor.

[0101] One or more controllers 936, which can include one or more system on a chip (SoC) 904 Figure 9C ) and / or one or more GPUs, can provide signals (e.g., representing commands) to one or more components and / or systems of the vehicle 900. For example, the one or more controllers can send signals to operate the vehicle brakes via one or more braking actuators 948, to operate the steering system 954 via one or more steering actuators 956, to operate the propulsion system 950 via one or more throttle / accelerator 952. The one or more controllers 936 can include one or more on-board (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to allow for autonomous driving and / or to assist a human driver in driving the vehicle 900. The one or more controllers 936 can include a first controller 936 for autonomous driving functionality, a second controller 936 for functional safety functionality, a third controller 936 for artificial intelligence functionality (e.g., computer vision), a fourth controller 936 for infotainment functionality, a fifth controller 936 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 936 can handle two or more of the above functionalities, two or more controllers 936 can handle a single functionality, and / or any combination thereof.

[0102] One or more controllers 936 can provide signage for controlling one or more components and / or systems of vehicle 900 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data can be received from, for example and without limitation, a global navigation satellite system (“GNSS”) sensor 958 (e.g., a global positioning system sensor), a RADAR sensor 960, an ultrasonic sensor 962, a LiDAR sensor 964, an inertial measurement unit (IMU) sensor 966 (e.g., an accelerometer, a gyroscope, a magnetic compass, a magnetometer, etc.), a microphone 996, a stereo camera 968, a wide-angle camera 970 (e.g., a fisheye camera), an infrared camera 972, a surround camera 974 (e.g., a 360-degree camera), a long and / or medium range camera 998, a speed sensor 944 (e.g., for measuring the speed of vehicle 900), a vibration sensor 942, a steering sensor 940, a brake sensor 946 (e.g., as part of a brake sensor system 946), and / or other sensor types.

[0103] One or more of controllers 936 can receive inputs from an instrument cluster 932 of vehicle 900 (e.g., represented by input data) and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 934, an audible annunciator, a speaker, and / or via other components of vehicle 900. These outputs can include information such as vehicle speed, velocity, time, map data (e.g., an “HD” map 922 of Figure 9C

[0104] ​The vehicle 900 also includes a network interface 924 that can communicate over one or more networks using one or more wireless antennas 926 and / or modems. For example, the network interface 924 can be capable of communicating over Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multi-Carrier (“CDMA2000”), and / or the like. The one or more wireless antennas 926 can also enable communication between objects (e.g., vehicles, mobile devices, and / or the like) in an implementation environment using one or more local area networks such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, and / or the like and / or one or more low power wide area networks (LPWANs) such as LoRaWAN, SigFox, and / or the like.

[0105] Figure 9B For example autonomous vehicle 900 for Figure 9A An example of camera locations and fields of view for the example autonomous vehicle 900 according to some embodiments of the present disclosure. The cameras and respective fields of view are one example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras can be included and / or these cameras can be located at different locations on the vehicle 900.

[0106] Camera types for the cameras can include, but are not limited to, digital cameras that can be suitable for use with components and / or systems of the vehicle 900. The cameras can operate at Automotive Safety Integrity Level (ASIL) B and / or at another ASIL. The camera types can have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, and / or the like, depending on the embodiment. The cameras can be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, a color filter array can include a red-white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras such as cameras with a

[0107] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more (e.g., all) of the cameras can simultaneously record and provide image data (e.g., video).

[0108] One or more of the cameras can be mounted in mounting assemblies such as custom designed (three-dimensional ("3D") printed) assemblies in order to cut off stray light and reflections from within the car (such as reflections of reflections from the dashboard off the windshield mirror) that can interfere with the image data capture capabilities of the cameras. With respect to wing mirror mounting assemblies, the wing mirror assemblies can be custom 3D printed such that the camera mounting plates match the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side view cameras, one or more cameras can also be integrated into the four pillars of each corner of the cab.

[0109] Cameras with fields of view that include the portion of the environment in front of the vehicle 900 (e.g., front-facing cameras) can be used for surround view to help identify the forward path and obstacles, and to assist in providing information critical to generating an occupancy grid and / or determining a preferred vehicle path with the help of one or more controllers 936 and / or control SoCs. Front-facing cameras can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. Front-facing cameras can also be used for ADAS functions and systems, including lane departure warning ("LDW"), adaptive cruise control ("ACC"), and / or other functions such as traffic sign recognition.

[0110] A wide variety of cameras can be used in the front-facing configuration, including, for example, monocular camera platforms including a complementary metal-oxide-semiconductor ("CMOS") color imager. Another example can be a wide-angle camera 970, which can be used to perceive objects (e.g., pedestrians, intersection traffic, or bicycles) entering the field of view from the periphery. Although Figure 9B Although only one wide-angle camera is illustrated in FIG. 9, there can be any number (including zero) of wide-angle cameras 970 on the vehicle 900. In addition, any number of long-range cameras 998 (e.g., long-view stereo camera pairs) can be used for depth-based object detection, especially for objects for which a neural network has not been trained. Long-range cameras 998 can also be used for object detection and classification and basic object tracking.

[0111] Any number of stereo cameras 968 can also be included in the front-mounted configuration. In at least one embodiment, one or more stereo cameras 968 may include an integrated control unit that includes a scalable processing unit that can provide a multi-core microprocessor and programmable logic (“FPGA”) with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternative stereo cameras 968 may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip that can measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 968 may be used in addition to those described herein or alternatively.

[0112] Cameras with a field of view including the side portion of the vehicle 900 (e.g., side-view cameras) can be used for surround view, providing information for creating and updating occupancy grids and generating side-impact collision warnings. For example, surround camera 974 (e.g., ... Figure 9B The four surround cameras 974 shown can be mounted on the vehicle 900. The surround cameras 974 can include wide-angle cameras 970, fisheye cameras, 360-degree cameras, and / or similar devices. Four examples are provided; the four fisheye cameras can be positioned at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 974 (e.g., left, right, and rear) and can utilize one or more other cameras (e.g., forward-facing cameras) as a fourth surround-view camera.

[0113] A camera with a field of view that includes the environment behind the vehicle 900 (e.g., a rear-view camera) can be used for parking assistance, surround view, rear collision warning, and creating and updating occupancy grids. A wide variety of cameras can be used, including but not limited to those also suitable as front-facing cameras as described herein (e.g., long-range and / or mid-range camera 998, stereo camera 968, infrared camera 972, etc.).

[0114] Figure 9C For use in accordance with some embodiments of this disclosure Figure 9AFIG. 1 is a block diagram of an example system architecture of an example autonomous vehicle 900. It should be understood that this arrangement and other arrangements described herein are set forth merely as examples. Arrangements other than those shown and other elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be omitted entirely. Further, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combinations and locations. Various functions described herein as being performed by an entity can be implemented in hardware, firmware, and / or software. For instance, various functions can be implemented by a processor executing instructions stored in a memory.

[0115] Figure 9C Each of the components, features, and systems of vehicle 900 are illustrated as being connected via bus 902. Bus 902 can include a controller area network (CAN) data interface (alternatively referred to herein as a "CAN bus"). The CAN can be a network within vehicle 900 that is used to assist in controlling various features and functions of vehicle 900, such as driving of brakes, acceleration, braking, steering, windshield wipers, etc. The CAN bus can be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus can be read to find steering wheel angle, ground speed, revolutions per minute (RPM) of the engine, button positions, and / or other vehicle status indicators. The CAN bus can be ASIL B compliant.

[0116] Although bus 902 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet can be used in addition to or instead of a CAN bus. Further, although bus 902 is represented with a single line, this is not intended to be limiting. For example, any number of buses 902 can be present, which can include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses that use different protocols. In some examples, two or more buses 902 can be used to perform different functions, and / or can be used for redundancy. For example, a first bus 902 can be used for collision avoidance functions, and a second bus 902 can be used for driving control. In any example, each bus 902 can be in communication with any component of vehicle 900, and two or more buses 902 can be in communication with the same components. In some examples, each SoC 1004, each controller 936, and / or each computer within the vehicle can have access to the same input data (e.g., inputs from sensors of vehicle 900), and can be connected to a common bus, such as a CAN bus.

[0117] The vehicle 900 can include one or more controllers 936, such as those described herein with respect to Figure 9A The controllers 936 can be used for a wide variety of functions. The controllers 936 can be coupled to any of the other distinct components and systems of the vehicle 900 and can be used for control of the vehicle 900, artificial intelligence of the vehicle 900, infotainment for the vehicle 900, and / or the like.

[0118] The vehicle 900 can include one or more system on chips (SoCs) 904. The SoCs 904 can include CPUs 906, GPUs 908, processors 910, caches 912, accelerators 914, data stores 916, and / or other components and features not illustrated. The SoCs 904 can be used to control the vehicle 900 in a wide variety of platforms and systems. For example, one or more SoCs 904 can be used in systems, such as systems of the vehicle 900, in conjunction with HD maps 922 that can obtain map refreshes and / or updates from one or more servers (such as the one or more servers 978) via a network interface 924. Figure 9D

[0119] The CPU 906 can include a CPU cluster or CPU complex (alternatively referred to herein as a “CCPLEX”). The CPU 906 can include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 906 can include eight cores in a coherent multi-processor configuration. In some embodiments, the CPU 906 can include four dual-core clusters with each cluster having a dedicated L2 cache (such as a 2 MB L2 cache). The CPU 906 (e.g., the CCPLEX) can be configured to support simultaneous cluster operation such that any combination of clusters of the CPU 906 can be active at any given time.

[0120] The CPU 906 can implement power management capabilities including one or more of the following features: individual hardware blocks can be automatically clock-gated when idle to save dynamic power; each core clock can be gated when the core is not actively executing instructions due to execution of WFI / WFE instructions; each core can be independently power-gated; each core cluster can be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster can be independently power-gated when all cores are power-gated. The CPU 906 can further implement an enhanced algorithm for managing power states in which the allowed power states and the desired wake-up time are specified and the hardware / microcode determines the best power state for the core, cluster, and CCPLEX to enter. The processing core can support a simplified power state entry sequence in software, with the work being offloaded to microcode. ​

[0121] GPU 908 can include an integrated GPU (alternatively referred to herein as an “iGPU”). GPU 908 can be programmable and efficient for parallel workloads. In some examples, GPU 908 can use an enhanced tensor instruction set. GPU 908 can include one or more streaming microprocessors, where each streaming microprocessor can include an LI cache (e.g., an LI cache having at least 96 KB of storage capacity), and two or more of the streaming microprocessors can share an L2 cache (e.g., an L2 cache having 512 KB of storage capacity). In some embodiments, GPU 908 can include at least eight streaming microprocessors. GPU 908 can use a compute application programming interface (API). In addition, GPU 908 can use one or more parallel computing platforms and / or programming models (e.g., NVIDIA’s CUDA).

[0122] In the case of automotive and embedded uses, GPU 908 can be power-optimized for best performance. For example, GPU 908 can be fabricated on a fin field-effect transistor (FinFET). However, this is not intended to be limiting, and GPU 908 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a thread warp scheduler, a dispatch unit, and / or a 64 KB register file. In addition, the streaming microprocessor can include independent parallel integer and floating point data paths to exploit the mix of computation and addressing computation to provide efficient execution of workloads. The streaming microprocessor can include independent thread scheduling capabilities to allow for more fine-grained synchronization and cooperation between parallel threads. The streaming microprocessor can include a combined LI data cache and shared memory unit to facilitate both simplified programming and performance.

[0123] GPU 908 can include a high-bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem that provides approximately 900 GB / s of peak memory bandwidth in some examples. In some examples, in addition to or alternatively from HBM memory, synchronous graphics random access memory (SGRAM) can be used, such as fifth generation graphics double data rate synchronous random access memory (GDDR5).

[0124] GPU 908 can include a unified memory technology that includes access counters to allow memory pages to be migrated more precisely to the processors that access them most frequently, improving efficiency of memory ranges shared between processors. In some examples, address translation services (ATS) support can be used to allow GPU 908 to access CPU 906 page tables directly. In such examples, when a GPU 908 memory management unit (MMU) experiences a miss, an address translation request can be transmitted to CPU 906. In response, CPU 906 can look up a virtual-to-physical mapping for the address in its page tables and transmit the translation back to GPU 908. In this way, the unified memory technology can allow a single unified virtual address space for memory of both CPU 906 and GPU 908, simplifying GPU 908 programming and porting applications to GPU 908.

[0125] Further, GPU 908 can include access counters that can track how frequently GPU 908 accesses other processors’ memory. The access counters can help ensure that memory pages are migrated to the physical memory of the processor that accesses these pages most frequently.

[0126] SoC 904 can include any number of caches 912, including those described herein. For example, caches 912 can include an L3 cache available to both CPU 906 and GPU 908 (e.g., connected to both CPU 906 and GPU 908). Caches 912 can include a write-back cache that can track the state of a line, for example, by using a cache coherency protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache can include 4 MB or more, although smaller cache sizes can also be used.

[0127] SoC 904 can include an arithmetic logic unit (ALU) that can be utilized in processing to perform any of a variety of tasks or operations with respect to vehicle 900, such as processing a DNN. Further, SoC 904 can include a floating point unit (FPU) (or other mathematical co-processor or digital co-processor type) for performing mathematical operations within the system. For example, SoC 904 can include one or more FPUs integrated as execution units within CPU 906 and / or GPU 908.

[0128] The SoC 904 can include one or more accelerators 914 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC 904 can include a hardware accelerator cluster that can include optimized hardware accelerators and / or a large on-chip memory. The large on-chip memory (e.g., 4MB SRAM) can enable the hardware accelerator cluster to accelerate neural networks and other computations. The hardware accelerator cluster can be used to supplement the GPU 908 and offload some of the tasks of the GPU 908 (e.g., freeing up more cycles of the GPU 908 for performing other tasks). As one example, the accelerators 914 can be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to be easily controlled for acceleration. As used herein, the term “CNN” can include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0129] The accelerators 914 (e.g., the hardware accelerator cluster) can include a deep learning accelerator (DLA). The DLA can include one or more tensor processing units (TPUs) that can be configured to provide an additional 100 billion operations per second for deep learning applications and inferencing. The TPUs can be accelerators that are configured to perform and optimized for performing image processing functions (e.g., for CNNs, RCNNs, etc.). The DLA can be further optimized for a specific set of neural network types and floating point operations and inferencing. The design of the DLA can provide higher performance per mm than general purpose GPUs and far exceeds the performance of CPUs. The TPUs can perform several functions, including single instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, for example, and post-processor functions.

[0130] The DLA can perform neural networks, especially CNNs, on processed or unprocessed data for any of a wide variety of functions, such as and not limited to: CNNs for object recognition and detection using data from a camera sensor; CNNs for distance estimation using data from a camera sensor; CNNs for emergency vehicle detection and identification and detection using data from a microphone; CNNs for face recognition and vehicle owner identification using data from a camera sensor; and / or CNNs for safety and / or safety related events.

[0131] The DLA can perform any of the functions of the GPU 908 and, by using an inferencing accelerator, for example, a designer can target the DLA or the GPU 908 for any function. For example, the designer can focus the processing and floating point operations of a CNN on the DLA and leave other functions to the GPU 908 and / or other accelerators 914.

[0132] Accelerator 914 (e.g., a cluster of hardware accelerators) can include a programmable vision accelerator (PVA), which can be alternatively referred to herein as a computer vision accelerator. The PVA can be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA can include, for example and without limitation, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0133] The RISC cores can interact with image sensors (e.g., image sensors of any of the cameras described herein), image landmark processors, and / or the like. Each of the RISC cores can include any number of memories. Depending on the embodiment, the RISC cores can use any of several protocols. In some examples, the RISC cores can execute a real-time operating system (RTOS). The RISC cores can be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores can include instruction caches and / or tightly coupled RAM.

[0134] The DMA can enable components of the PVA to access system memory independently of the CPU 906. The DMA can support any number of features to provide optimizations to the PVA, including, but not limited to, supporting multi-dimensional addressing and / or circular addressing. In some examples, the DMA can support addressing up to six or more dimensions, which can include block width, block height, block depth, horizontal block stride, vertical block stride, and / or depth stride.

[0135] The vector processors can be programmable processors that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide landmark processing capabilities. In some examples, the PVA can include a PVA core and two vector processing subsystem partitions. The PVA core can include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystems can operate as the main processing engines of the PVA and can include vector processing units (VPUs), instruction caches, and / or vector memories (e.g., VMEM). The VPU cores can include digital landmark processors, such as, for example, single instruction multiple data (SIMD), very long instruction word (VLIW) digital landmark processors. The combination of SIMD and VLIW can enhance throughput and rate.

[0136] Each of the vector processors can include an instruction cache and can be coupled to a dedicated memory. As a result, in some examples, each of the vector processors can be configured to execute independently of the other vector processors. In other examples, the vector processors included in a particular PVA can be configured to employ data parallelization. For example, in some embodiments, multiple vector processors included in a single PVA can execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can execute different computer vision algorithms on the same image simultaneously, or even different algorithms on a sequence of images or portions of an image. Any number of PVAs can be included in a hardware accelerator cluster, and any number of vector processors can be included in each of those PVAs, among other things. Furthermore, the PVAs can include additional error-correcting code (ECC) memory to enhance overall system security.

[0137] The accelerator 914 (e.g., hardware accelerator cluster) can include an on-chip computer vision network and SRAM to provide high bandwidth, low latency SRAM for the accelerator 914. In some examples, the on-chip memory can include at least 4 MB of SRAM composed of, for example and without limitation, eight field-programmable memory blocks, which can be accessed by both the PVA and the DLA. Each pair of memory blocks can include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory can be used. The PVA and the DLA can access the memory via a backbone that provides high-speed memory access to the PVA and the DLA. The backbone can include an on-chip computer vision network that interconnects the PVA and the DLA to the memory, for example using an APB.

[0138] The on-chip computer vision network can include an interface that determines that both the PVA and the DLA provide ready and valid flags before transmitting any control flags / addresses / data. Such an interface can provide separate phases and separate channels for transmitting control flags / addresses / data, as well as burst communications for continuous data transmission. This type of interface can comply with ISO 26262 or IEC 615010 standards, but other standards and protocols can also be used.

[0139] In some examples, the SoC 904 can include a real-time ray tracing hardware accelerator, such as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator can be used to quickly and efficiently determine locations and extents of objects (e.g., within a world model) in order to generate real-time visualizations simulations for RADAR signature interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulation, for general wave propagation simulation, for comparison to LiDAR data for purposes of localization and / or other functionality, and / or for other uses. In some embodiments, one or more tree traversal units (TTUs) can be used to perform one or more ray tracing related operations.

[0140] The accelerator 914 (e.g., hardware accelerator cluster) has a wide range of autonomous driving uses. The PVA can be a programmable vision accelerator that can be used for key processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are a good match for algorithm domains that require predictable processing, low power, and low latency. In other words, the PVA performs well on semi-dense or dense regular computations, and even on small data sets that require predictable runtimes with low latency and low power. Thus, in the context of a platform for autonomous vehicles, the PVA is designed to run classical computer vision algorithms because they are effective at object detection and integer math operations.

[0141] For example, according to one embodiment of the technology, the PVA is used to perform computer stereo vision. In some examples, a semi-global matching based algorithm can be used, although this is not intended to be limiting. Many applications for level 3-5 autonomous driving require instant motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.). The PVA can perform computer stereo vision functions on input from two monocular cameras.

[0142] In some examples, the PVA can be used to perform dense optical flow. Raw RADAR data is processed according to a process (e.g., using a 4D fast Fourier transform) to provide processed RADAR. In other examples, the PVA is used for time-of-flight depth processing, such as by processing raw time-of-flight data to provide processed time-of-flight data.

[0143] The DLA can be used to run any type of network to enhance control and driving safety, including, for example, a neural network that outputs a confidence metric for each object detection. Such a confidence value can be interpreted as a probability, or as providing a relative "weight" for each detection compared to other detections. The confidence value enables the system to make further decisions about which detections should be considered true positive detections and not false positive detections. For example, the system can set a threshold for confidence, and only consider detections that exceed the threshold as true positive detections. In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically perform an emergency brake, which is obviously undesirable. Thus, only the most confident detections should be considered a trigger for AEB. The DLA can run a neural network for regression of a confidence value. The neural network can take as its input at least some subset of parameters, such as a bounding box dimension, a ground plane estimate obtained (e.g., from another subsystem), inertial measurement unit (IMU) sensor 966 outputs related to vehicle 900 orientation, distance, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LiDAR sensor 964 or RADAR sensor 960), etc.

[0144] SoC 904 can include one or more data stores 916 (e.g., memory). Data stores 916 can be on-chip memory of SoC 904, which can store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and safety, data stores 916 can be large enough in capacity to store multiple instances of a neural network. Data stores 916 can include L2 or L3 cache 912. References to data stores 916 can include references to memory associated with PVAs, DLAs, and / or other accelerators 914 as described herein.

[0145] The SoC 904 can include one or more processors 910 (e.g., embedded processors). The processors 910 can include a boot and power management processor, which can be a specialized processor and subsystem for handling boot power and management functions and related security implementations. The boot and power management processor can be part of the SoC 904 boot sequence and can provide run-time power management services. The boot power and management processor can provide clock and voltage programming, auxiliary system low power state transitions, SoC 904 thermal and temperature sensor management, and / or SoC 904 power state management. Each temperature sensor can be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC 904 can use the ring oscillator to detect the temperature of the CPU 906, GPU 908, and / or accelerator 914. If it is determined that the temperature exceeds a threshold, the boot and power management processor can enter a temperature fault routine and place the SoC 904 in a lower power state and / or place the vehicle 900 in a driver safe park mode (e.g., safely park the vehicle 900).

[0146] The processors 910 can also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio over multiple interfaces and a range of widely flexible audio I / O interfaces. In some examples, the audio processing engine is a specialized processor core with a digital signature processor with dedicated RAM.

[0147] The processors 910 can also include an always-on processor engine that can provide the necessary hardware features to support low power sensor management and wake-up use cases. The always-on processor engine can include a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0148] The processors 910 can also include a security cluster engine that includes a specialized processor subsystem that handles security management for automotive applications. The security cluster engine can include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In a secure mode, the two or more cores can operate in a lockstep mode and act as a single core with comparison logic that detects any differences between their operations.

[0149] The processors 910 can also include a real-time camera engine that can include a specialized processor subsystem for handling real-time camera management.

[0150] The processor 910 can further include a high dynamic range flag processor, which can include an image flag processor, which is a hardware engine that is part of the camera processing pipeline.

[0151] The processor 910 can include a video image compositor, which can be a processing block (e.g., implemented on a microprocessor), that implements video post-processing functions needed by a video playback application to produce the final image for the player window. The video image compositor can perform lens distortion correction on the wide-angle camera 970, surround camera 974, and / or on the cab-in monitor camera sensors. The cab-in monitor camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to identify cab-in events and respond accordingly. The cab-in system can perform lip reading to activate mobile phone services and place a call, dictate an email, change the vehicle destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode, and are disabled otherwise.

[0152] The video image compositor can include enhanced temporal noise reduction for spatial and temporal noise reduction. For example, where motion is present in the video, the noise reduction appropriately weights the spatial information, reducing the weight of information provided by neighboring frames. Where the image or portions of the image do not include motion, the temporal noise reduction performed by the video image compositor can use information from previous images to reduce noise in the current image.

[0153] The video image compositor can also be configured to perform stereo correction on input stereo lens frames. The video image compositor can further be used for user interface composition when the operating system desktop is in use and the GPU 908 does not need to continuously render new surfaces. Even when the GPU 908 is powered on and active, doing 3D rendering, the video image compositor can be used to offload the GPU 908 to improve performance and responsiveness.

[0154] The SoC 904 can further include a Mobile Industry Processor Interface (MIPI) camera serial interface for receiving video and input from cameras, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functions. The SoC 904 can further include an input / output controller that can be controlled by software and can be used to receive I / O flags that are not committed to a particular role.

[0155] The SoC 904 can also include a wide range of peripheral device interfaces to allow communication with peripherals, audio codecs, power management, and / or other devices. The SoC 904 can be used to process data from cameras (connected over Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LiDAR sensor 964, RADAR sensor 960, etc. that can be connected over Ethernet), data from the bus 902 (e.g., speed of the vehicle 900, steering wheel position, etc.), data from the GNSS sensor 958 (connected over Ethernet or CAN bus). The SoC 904 can also include a dedicated high-performance mass storage controller, which can include their own DMA engine, and which can be used to free the CPU 906 from routine data management tasks.

[0156] The SoC 904 can be an end-to-end platform with a flexible architecture that spans automation levels 3-5, providing an integrated functional safety architecture for a platform that leverages and efficiently uses computer vision and ADAS technology to achieve diversity and redundancy, along with deep learning tools. The SoC 904 can be faster, more reliable, and even more energy and space efficient than conventional systems. For example, the accelerator 914, when combined with the CPU 906, GPU 908, and data storage 916, can provide a fast and efficient platform for level 3-5 autonomous vehicles.

[0157] The technology thus provides capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on CPUs that can be configured using high-level programming languages such as the C programming language to perform a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to, for example, execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for on-board ADAS applications and a requirement for practical level 3-5 autonomous vehicles.

[0158] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a cluster of hardware accelerators, the technology described herein allows multiple neural networks to be executed simultaneously and / or sequentially, and the results to be combined together to achieve level 3-5 autonomous driving functionality. For example, a CNN executed on a DLA or dGPU (e.g., GPU 920) can include text and word recognition, allowing a supercomputer to read and understand traffic signs, including signs for which a neural network has not been specifically trained. The DLA can also include a neural network that is able to recognize, interpret, and provide a semantic understanding of the sign, and pass that semantic understanding to a path planning module running on the CPU complex.

[0159] As another example, multiple neural networks can be run simultaneously as required for level 3, 4, or 5 driving. For example, a warning sign consisting of the words "Caution: flashing lights indicate icy conditions" along with electric lights can be interpreted by several neural networks independently or collectively. The sign itself can be recognized by a first deployed neural network (e.g., a trained neural network) as a traffic sign, the text "flashing lights indicate icy conditions" can be interpreted by a second deployed neural network that informs the vehicle's path planning software (preferably executing on the CPU complex) that icy conditions exist when flashing lights are detected. The flashing lights can be recognized by operating a third deployed neural network over multiple frames that informs the vehicle's path planning software of the presence (or absence) of flashing lights. All three neural networks can be run simultaneously, for example, within the DLA and / or on the GPU 908.

[0160] In some examples, a CNN for face recognition and owner recognition can use data from the camera sensor to recognize the presence of an authorized driver and / or owner of the vehicle 900. A processing engine always on the sensor can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in a safe mode, disable the vehicle when the owner leaves the vehicle. In this way, the SoC 904 provides security against theft and / or carjacking.

[0161] In another example, a CNN for emergency vehicle detection and recognition can use data from the microphones 996 to detect and recognize emergency vehicle sirens. In contrast to conventional systems that detect sirens using a general classifier and manually extract features, the SoC 904 uses a CNN to classify ambient and urban sounds as well as to classify visual data. In a preferred embodiment, a CNN running on the DLA is trained to recognize the relative closing speed of an emergency vehicle (e.g., by using the Doppler effect). The CNN can also be trained to recognize emergency vehicles specific to the local area in which the vehicle is operating as recognized by the GNSS sensor 958. Thus, for example, when operating in the European Union, the CNN will seek to detect European Union sirens, and when in the United States, the CNN will seek to recognize sirens that are only North American. Once an emergency vehicle is detected, a control program can be used to execute an emergency vehicle safety routine that slows the vehicle, pulls over to the side of the road, stops the vehicle, and / or idles the vehicle until the emergency vehicle passes, with the assistance of the ultrasonic sensors 962.

[0162] The vehicle can include a CPU 918 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 904 via a high-speed interconnect (e.g., PCIe). The CPU 918 can include, for example, an X106 processor. The CPU 918 can be used to perform any of a wide variety of functions, including, for example, arbitrating potentially inconsistent results between ADAS sensors and the SoC 904, and / or monitoring the status and health of the controller 936 and / or infotainment SoC 930.

[0163] The vehicle 900 can include a GPU 920 (e.g., a discrete GPU or dGPU) that can be coupled to the SoC 904 via a high-speed interconnect (e.g., NVIDIA’s NVLINK). The GPU 920 can provide additional artificial intelligence functionality, for example, by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based at least in part on input (e.g., sensor data) from sensors of the vehicle 900.

[0164] The vehicle 900 can also include a network interface 924 that can include one or more wireless antennas 926 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 924 can be used to enable wireless connections over the Internet with a cloud (e.g., with a server 978 and / or other network devices), with other vehicles, and / or with computing devices (e.g., client devices of passengers). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across a network and through the Internet). The direct link can be provided using a car-to-car communication link. The car-to-car communication link can provide the vehicle 900 with information about vehicles that are approaching the vehicle 900 (e.g., vehicles in front of, to the side of, and / or behind the vehicle 900). This functionality can be part of a cooperative adaptive cruise control functionality of the vehicle 900.

[0165] The network interface 924 can include a SoC that provides modulation and demodulation functionality and enables the controller 936 to communicate over a wireless network. The network interface 924 can include a radio frequency front end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. The frequency conversion can be performed through well-known processes and / or can be performed using a super-heterodyne process. In some examples, the radio frequency front end functionality can be provided by a separate chip. The network interface can include wireless functionality for communication over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0166] The vehicle 900 can also include a data storage 928 that can include off-chip (e.g., off-SoC 904) storage. The data storage 928 can include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disks, and / or other components and / or devices that can store data for at least one bit. The storage 928

[0167] The vehicle 900 can also include a GNSS sensor 958. The GNSS sensor 958 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used to assist in mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 958 can be used, including, for example and without limitation, a GPS using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0168] The vehicle 900 can also include a RADAR sensor 960. The RADAR sensor 960 can be used by the vehicle 900 for long-range vehicle detection, even in darkness and / or adverse weather conditions. The RADAR functional safety level can be ASIL B. The RADAR sensor 960 can use the CAN and / or the bus 902 (e.g., to transmit data generated by the RADAR sensor 960) for control as well as access to object tracking data, in some examples, Ethernet for access to raw data. A wide variety of RADAR sensor types can be used. For example and without limitation, the RADAR sensor 960 can be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor is used.

[0169] The RADAR sensor 960 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In some examples, long-range RADAR can be used for adaptive cruise control functionality. Long-range RADAR systems can provide a wide field of view (e.g., 250 m range) implemented through two or more independent scans. The RADAR sensor 960 can help distinguish between static and moving objects, and can be used by the ADAS system for emergency brake assist and forward collision warning. The long-range RADAR sensor can include a single-station multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas, as well as a high-speed CAN and FlexRay interface. In examples with six antennas, the central four antennas can create focused beam patterns designed to record the surroundings of the vehicle 900 at higher speed with minimal traffic interference from adjacent lanes. The other two antennas can extend the field of view, making it possible to quickly detect vehicles entering or leaving the lane of the vehicle 900.

[0170] As one example, a mid-range RADAR system can include a range of up to 160 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 150 degrees (rear). A short-range RADAR system can include, but is not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor the rear and the blind spot next to the vehicle.

[0171] A short-range RADAR system can be used in an ADAS system for blind spot detection and / or lane change assist.

[0172] The vehicle 900 can also include ultrasonic sensors 962. The ultrasonic sensors 962, which can be placed on the front, rear, and / or sides of the vehicle 900, can be used for parking assist and / or to create and update an occupancy grid. A wide variety of ultrasonic sensors 962 can be used, and different ultrasonic sensors 962 can be used for different detection ranges (e.g., 2.5 m, 4 m). The ultrasonic sensors 962 can operate at an ASIL B functional safety level.

[0173] The vehicle 900 can include LIDAR sensors 964. The LiDAR sensors 964 can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LiDAR sensors 964 can be at an ASIL B functional safety level. In some examples, the vehicle 900 can include multiple LiDAR sensors 964 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0174] In some examples, the LiDAR sensors 964 can be capable of providing a list of objects and their distances for a 360-degree field of view. A commercially available LiDAR sensor 964 can have, for example, an advertised range of approximately 900 m, a precision of 2 cm - 3 cm, and support for a 900 Mbps Ethernet connection. In some examples, one or more flush LiDAR sensors 964 can be used. In such examples, the LiDAR sensors 964 can be implemented as small devices that can be embedded to the front, rear, sides, and / or corners of the vehicle 900. In such examples, the LiDAR sensors 964 can provide a field of view of up to 120 degrees horizontal and 35 degrees vertical with a range of 200 m, even for low reflectivity objects. Front-mounted LiDAR sensors 964 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0175] In some examples, LiDAR technology such as 3D Flash LiDAR can also be used. 3D Flash LiDAR uses a flash of laser light as a source of emission to illuminate the vehicle’s surroundings up to about 200 m. The Flash LiDAR unit includes a receptor that records the laser pulse transmission time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LiDAR can allow for the generation of highly accurate and distortion-free images of the surroundings with each laser flash. In some examples, four Flash LiDAR sensors can be deployed, one on each side of the vehicle 900. Available 3D Flash LiDAR systems include solid-state 3D staring array LiDAR cameras (e.g., non-scanning LiDAR devices) that have no moving parts other than fans. The Flash LiDAR device can use a 5-nanosecond Class I (eye-safe) laser pulse per frame and can capture the reflected laser light in the form of a 3D range point cloud and co-registered intensity data. By using Flash LiDAR, and because Flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 964 can be less susceptible to motion blur, vibration, and / or jostling.

[0176] The vehicle can also include an IMU sensor 966. In some examples, the IMU sensor 966 can be located at the center of the rear axle of the vehicle 900. The IMU sensor 966 can include, for example and without limitation, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in six-axis applications, the IMU sensor 966 can include an accelerometer and a gyroscope, while in nine-axis applications, the IMU sensor 966 can include an accelerometer, a gyroscope, and a magnetometer.

[0177] In some embodiments, the IMU sensor 966 can be implemented as a microelectromechanical systems (MEMS) inertial navigation system (INS) that combines a microelectromechanical systems (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor 966 can enable the vehicle 900 to estimate heading without input from a magnetic sensor by directly observing the change in velocity from GPS to the IMU sensor 966 and correlating it. In some examples, the IMU sensor 966 and the GNSS sensor 958 can be combined into a single integrated unit.

[0178] The vehicle can include a microphone 996 placed in and / or around the vehicle 900. The microphone 996 can be used for emergency vehicle detection and identification, among other things.

[0179] The vehicle can also include any number of camera types, including stereo cameras 968, wide-view cameras 970, infrared cameras 972, surround cameras 974, long and / or mid-range cameras 998, and / or other camera types. These cameras can be used to capture image data around the entire periphery of the vehicle 900. The types of cameras used depend on the embodiment and requirements of the vehicle 900, and any combination of camera types can be used to provide the necessary coverage around the vehicle 900. Further, the number of cameras can vary depending on the embodiment. For example, the vehicle can include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As one example and without limitation, the cameras can support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described in more detail herein with respect to Figure 9A and Figure 9B are described in more detail.

[0180] The vehicle 900 can also include vibration sensors 942. The vibration sensors 942 can measure vibrations of components of the vehicle, such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 942 are used, differences between the vibrations can be used to determine the friction or slip of the road surface (e.g., when there is a difference in vibration between a powered axle and a free spinning axle).

[0181] The vehicle 900 can include an ADAS system 938. In some examples, the ADAS system 938 can include a SoC. The ADAS system 938 can include adaptive / automatic / autonomous cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functionality.

[0182] The ACC system can use RADAR sensors 960, LiDAR sensors 964, and / or cameras. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately ahead of the vehicle 900 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Lateral ACC performs distance keeping and, if necessary, suggests a lane change for the vehicle 900. Lateral ACC is related to other ADAS applications such as LCA and CWS.

[0183] CACC uses information from other vehicles, which can be received from other vehicles via a wireless link via the network interface 924 and / or wireless antenna 926 or indirectly through a network connection, such as through the Internet. Direct links can be provided by a vehicle-to-vehicle (V2V) communication link, while indirect links can be an infrastructure-to-vehicle (I2V) communication link. Generally, the V2V communication concept provides information about the immediately preceding vehicles, such as vehicles immediately ahead of and in the same lane as the vehicle 900, while the I2V communication concept provides information about traffic further ahead. A CACC system can include either or both of I2V and V2V information sources. Given information about vehicles ahead of the vehicle 900, CACC can be more reliable, and it has the potential to improve traffic flow and reduce road congestion.

[0184] FCW systems are designed to alert the driver to a hazard so that the driver can take corrective action. FCW systems use a front-facing camera and / or RADAR sensor 960 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as displays, speakers, and / or vibrating components. FCW systems can provide warnings in the form of, for example, sound, visual warnings, vibrations, and / or quick brake pulses.

[0185] AEB systems detect an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within specified time or distance parameters. AEB systems can use a front-facing camera and / or RADAR sensor 960 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When an AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes in an effort to prevent or at least mitigate the effects of a predicted collision. AEB systems can include technologies such as dynamic brake support and / or crash imminent braking.

[0186] LDW systems provide visual, audible, and / or tactile warnings such as steering wheel or seat vibrations to alert the driver when the vehicle 900 is crossing lane markers. The LDW system is not activated when the driver indicates an intentional lane departure by activating a turn signal. LDW systems can use a front-side facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as displays, speakers, and / or vibrating components.

[0187] An LKA system is a variation of the LDW system. If the vehicle 900 begins to leave the lane, the LKA system provides a steering input or brake to correct the vehicle 900. A BSW system detects and warns the driver of vehicles in the car's blind spot. The BSW system can provide visual, audible, and / or tactile warnings to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses a turn signal.

[0188] A BSW system can use rear-side facing cameras and / or one or more RADAR sensors 960 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0189] A RCTW system can provide visual, audible, and / or tactile notifications when an object is detected outside the range of the rear-facing camera while the vehicle 900 is backing up. Some RCTW systems include AEB to ensure vehicle brakes are applied to avoid a collision. A RCTW system can use one or more rear-facing RADAR sensors 960 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0190] Conventional ADAS systems can be prone to false positive results, which can annoy and distract the driver, but typically are not catastrophic because the ADAS system alerts the driver and allows the driver to decide whether the safety condition is truly present and act accordingly. However, in an autonomous vehicle 900, in the case of conflicting results, the vehicle 900 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., the first controller 936 or the second controller 936). For example, in some embodiments, the ADAS system 938 can be a secondary and / or auxiliary computer for providing perception information to a backup computer plausibility module. The backup computer plausibility monitor can run redundant diverse software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 938 can be provided to a supervisory MCU. If the outputs from the primary computer and the secondary computer conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0191] In some examples, the host computer can be configured to provide a confidence score to the supervisory MCU indicating the host computer's confidence in the selected result. If the confidence score exceeds a threshold, then the supervisory MCU can follow the host computer's direction, regardless of whether the secondary computer provides conflicting or inconsistent results. In the event that the confidence score does not satisfy the threshold and in the event that the host computer and the secondary computer indicate different results (e.g., a conflict), the supervisory MCU can arbitrate between the computers to determine the appropriate result.

[0192] The supervisory MCU can be configured to run a neural network that is trained and configured to determine conditions under which the secondary computer provides false alarms based at least in part on the output from the host computer and the secondary computer. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer can be trusted and when it cannot. For example, when the secondary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying metal objects that are not in fact dangerous, such as drain grates or manhole covers that trigger false alarms. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to disregard the LDW when a cyclist or pedestrian is present and lane departure is in fact the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU can include at least one of a DLA or a GPU suitable for running a neural network with associated memory. In preferred embodiments, the supervisory MCU can include and / or be included as a component of the SoC 904.

[0193] In other examples, the ADAS system 938 can include a secondary computer that performs ADAS functions using traditional computer vision rules. In this way, the secondary computer can use classic computer vision rules (if-then), and the presence of a neural network in the supervisory MCU can improve reliability, safety, and performance. For example, the diverse implementation and intentional non-identity make the overall system more fault-tolerant, especially with respect to faults caused by software (or software-hardware interface) functions. For example, if there is a software bug or error in the software running on the host computer and the non-identical software code running on the secondary computer provides the same overall result, then the supervisory MCU can be more confident that the overall result is correct and that the bug in the software or hardware on the host computer did not cause a substantial error.

[0194] In some examples, the output of the ADAS system 938 can be fed to a perception block of the host computer and / or a dynamic driving task block of the host computer. For example, if the ADAS system 938 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information in identifying the object. In other examples, the secondary computer can have its own neural network that is trained and thus reduces the risk of false positives as described herein.

[0195] The vehicle 900 can also include an infotainment SoC 930 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as a SoC, the infotainment system can not be a SoC and can include two or more discrete components. The infotainment SoC 930 can include a combination of hardware and software that can be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephony (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.), and / or information services (e.g., navigation system, park assist, radio data system, vehicle related information such as fuel level, total distance covered, brake fuel level, oil level, doors open / closed, air filter information, etc.) to the vehicle 900. For example, the infotainment SoC 930 can include a radio, a disc player, a navigation system, a video player, USB and Bluetooth connectivity, an in-car computer, in-car entertainment, WiFi, steering wheel audio controls, hands-free voice controls, a heads-up display (HUD), the HMI display 934, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 930 can further be used to provide information (e.g., visual and / or audible) to a user of the vehicle, such as information from the ADAS system 938, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0196] The infotainment SoC 930 can include GPU functionality. The infotainment SoC 930 can communicate with other devices, systems, and / or components of the vehicle 900 over the bus 902 (e.g., a CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 930 can be coupled to a supervisory MCU such that, in the event of a failure of the host controller 936 (e.g., a primary and / or backup computer of the vehicle 900), the GPU of the infotainment system can perform some autonomous driving functions. In such examples, the infotainment SoC 930 can place the vehicle 900 in a driver safe park mode as described herein.

[0197] The vehicle 900 can also include an instrument cluster 932 (e.g., a digital instrument cluster, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 932 can include a controller and / or supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 932 can include a set of instruments, such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seat belt warning light, parking brake warning light, engine malfunction light, supplemental restraint system (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information can be displayed and / or shared between the infotainment SoC 930 and the instrument cluster 932. In other words, the instrument cluster 932 can be included as part of the infotainment SoC 930, or vice versa.

[0198] Figure 9D FIG. 10 illustrates a system diagram of communication between a cloud-based server and an example autonomous vehicle 900 in accordance with some embodiments of the present disclosure. Figure 9A FIG. 10 illustrates a system diagram of communication between a cloud-based server and an example autonomous vehicle 900 in accordance with some embodiments of the present disclosure. The system 1076 can include servers 978, a network 1090, and vehicles including the vehicle 900. The servers 978 can include a plurality of GPUs 1084(A)-1084(H) (collectively referred to herein as GPUs 1084), PCIe switches 1082(A)-1082(D) (collectively referred to herein as PCIe switches 1082), and / or CPUs 1080(A)-1080(B) (collectively referred to herein as CPUs 1080). The GPUs 1084, CPUs 1080, and PCIe switches can be interconnected with high-speed interconnects such as, for example and without limitation, NVLink interfaces 1088 developed by NVIDIA and / or PCIe connections 1086. In some examples, the GPUs 1084 are connected via NVLink and / or NVSwitch SoC, and the GPUs 1084 and PCIe switches 1082 are connected via PCIe interconnects. Although eight GPUs 1084, two CPUs 1080, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the servers 978 can include any number of GPUs 1084, CPUs 1080, and / or PCIe switches. For example, each of the servers 978 can include eight, sixteen, thirty-two, and / or more GPUs 1084.

[0199] The server 978 can receive image data from vehicles over the network 1090 and representing images showing unexpected or changing road conditions such as a recently started road work. The server 978 can transmit neural networks 1092, updated neural networks 1092, and / or map information 1094, including information about traffic and road conditions, to vehicles over the network 1090. Updates to the map information 1094 can include updates to the HD map 922, e.g., information about construction sites, potholes, curves, flooding, or other obstacles. In some examples, the neural networks 1092, updated neural networks 1092, and / or map information 1094 can have been produced from experience using training performed at a data center (e.g., using the server 978 and / or other servers) and / or from data received from any number of vehicles in the environment.

[0200] The server 978 can be used to train machine learning models (e.g., neural networks) based on training data. The training data can be generated by vehicles and / or can be generated in simulations (e.g., using game engines). In some examples, the training data is labeled (e.g., in cases where the neural network benefits from supervised learning) and / or undergoes other pre-processing, while in other examples, the training data is not labeled and / or pre-processed (e.g., in cases where the neural network does not require supervised learning). The training can be performed according to any one or more categories of machine learning techniques, including but not limited to categories such as: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including spare dictionary learning), rule-based machine learning, anomaly detection, and any variants or combinations thereof. Once the machine learning models are trained, the machine learning models can be used by vehicles (e.g., transmitted to vehicles over the network 1090), and / or the machine learning models can be used by the server 978 to remotely monitor vehicles.

[0201] In some examples, the server 978 can receive data from vehicles and apply the data to the latest real-time neural networks for real-time intelligent inference. The server 978 can include deep learning supercomputers and / or specialized AI computers supported by GPUs 1084, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, the server 978 can include deep learning infrastructure of a data center powered by CPUs only.

[0202] The deep learning infrastructure of the server 978 can be capable of fast real-time inference, and can use this capability to assess and validate the health of the processors, software, and / or associated hardware in the vehicle 900. For example, the deep learning infrastructure can receive periodic updates from the vehicle 900, such as a sequence of images and / or objects located in the sequence of images that the vehicle 900 has located (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify the objects and compare them to the objects identified by the vehicle 900, and if the results do not match and the infrastructure concludes that the AI in the vehicle 900 is malfunctioning, the server 978 can transmit a flag to the vehicle 900 instructing the failsafe computer of the vehicle 900 to take control, notify the passengers, and complete a safe parking operation.

[0203] For inference, the server 978 can include GPUs 1084 and one or more programmable inference accelerators (such as NVIDIA’s TensorRT). The combination of GPU-powered servers and inference-accelerated can make real-time response possible. In other examples, such as where performance is less important, CPU-, FPGA-, and other processor-powered servers can be used for inference.

[0204] Example Computing Device

[0205] Figure 10 is a block diagram of an example computing device 1000 suitable for implementing some embodiments of the present disclosure. The computing device 1000 can include an interconnection system 1002 coupling the following devices: a memory 1004, one or more central processing units (CPUs) 1006, one or more graphics processing units (GPUs) 1008, a communication interface 1010, input / output (I / O) ports 1012, I / O components 1014, a power supply 1016, one or more presentation components 1018 (e.g., one or more displays), and one or more logic units 1020. In at least one embodiment, the one or more computing devices 1000 can include one or more virtual machines (VMs), and / or any component thereof can include virtual components (e.g., virtual hardware components). For non-limiting examples, one or more of the GPUs 1008 can include one or more vGPUs, one or more of the CPUs 1006 can include one or more vCPUs, and / or one or more of the logic units 1020 can include one or more virtual logic units. As such, the one or more computing devices 1000 can include discrete components (e.g., a full GPU dedicated to the computing device 1000), virtual components (e.g., a portion of a GPU dedicated to the computing device 1000), or a combination thereof.

[0206] Although Figure 10 various blocks of the computing device are shown as being connected by the interconnect system 1002, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component 1018, such as a display device, can be considered an I / O component 1014 (e.g., if the display is a touch screen). As another example, the CPU 1006 and / or GPU 1008 can include memory (e.g., the memory 1004 can represent a storage device in addition to the memory of the GPU 1008, CPU 1006, and / or other components). Thus, Figure 10 the computing device of is merely illustrative. Distinction is not made between a “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “hand-held device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types within the scope of the computing device of Figure 10 .

[0207] The interconnect system 1002 can represent one or more links or buses, such as an address bus, data bus, control bus, or a combination thereof. The interconnect system 1002 can include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU 1006 can be directly connected to the memory 1004. Further, the CPU 1006 can be directly connected to the GPU 1008. Where there are direct or point-to-point connections between components, the interconnect system 1002 can include a PCIe link to perform the connection. In these examples, a PCI bus need not be included in the computing device 1000.

[0208] The memory 1004 can include any of a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device 1000. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media.

[0209] Computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, and / or other data types. For example, memory 1004 can store computer readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media can include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computing device 1000. As used herein, computer storage media does not include signals per se.

[0210] Computer storage media can embody computer readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" can refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.

[0211] The CPUs 1006 can be configured to execute at least some of the computer readable instructions to control one or more components of the computing device 1000 to perform one or more of the methods and / or processes described herein. The CPUs 1006 can each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling multiple software threads concurrently. The CPUs 1006 can include any type of processors and can include different types of processors depending on the type of computing device 1000 being implemented (e.g., a mobile device having fewer cores for a processor and a server having more cores for a processor). For example, depending on the type of computing device 1000, the processor can be an Advanced RISC Machines (ARM) processor implemented using a Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 1000 can include one or more CPUs 1006 in addition to, or as an alternative to, one or more microprocessors or co-processors such as mathematical co-processors.

[0212] One or more GPUs 1008, in addition to or in place of one or more CPUs 1006, can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 1000 to perform one or more of the methods and / or processes described herein. One or more of GPUs 1008 can be integrated GPUs (e.g., with one or more of CPUs 1006) and / or one or more of GPUs 1008 can be discrete GPUs. In embodiments, one or more of GPUs 1008 can be a co-processor of one or more of CPUs 1006. GPUs 1008 can be used by computing device 1000 to render graphics (e.g., 3D graphics) or to perform general purpose computing. For example, GPUs 1008 can be used for general purpose computing on GPUs (GPGPU). GPUs 1008 can include hundreds or thousands of cores capable of handling hundreds or thousands of software threads concurrently. GPUs 1008 can generate pixel data for output images in response to rendering commands (e.g., received from CPUs 1006 via a host interface). GPUs 1008 can include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory can be included as part of memory 1004. GPUs 1008 can include two or more GPUs operating in parallel (e.g., via a link). The link can connect the GPUs directly (e.g., using NVLINK) or can connect the GPUs through a switch (e.g., using NVSwitch). When combined together, each GPU 1008 can generate pixel data or GPGPU data for a different portion of an output or for a different output (e.g., a first GPU for a first image and a second GPU for a simulated image). Each GPU can include its own memory or can share memory with other GPUs.

[0213] In addition to or in place of CPU(s) 1006 and / or GPU(s) 1008, logic units 1020 can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 1000 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPU(s) 1006, one or more GPU(s) 1008, and / or one or more logic units 1020 can perform any combination of the methods, processes, and / or portions thereof, discretely or jointly. One or more of logic units 1020 can be part of one or more of CPU(s) 1006 and / or integrated in one or more of CPU(s) 1006 and / or one or more of logic units 1020 can be discrete components or otherwise external to CPU(s) 1006 and / or GPU(s) 1008. In embodiments, one or more of logic units 1020 can be a co-processor of one or more of CPU(s) 1006 and / or one or more of GPU(s) 1008.

[0214] Examples of logic units 1020 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), a tensor core (TC), a tensor processing unit (TPU), a pixel vision core (PVC), a vision processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multi-processor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), a programmable vision accelerator (PVA) (which can include one or more direct memory access (DMA) systems), one or more vision or vector processing units (VPUs), one or more pixel processing engines (PPEs) (e.g., including a 2D array of processing elements each in north, south, east, west communication with one or more other processing elements in the array), one or more decoupled accelerators or units (e.g., a decoupled lookup table (DLUT) accelerator or unit), etc., a vision processing unit (VPU), an optical flow accelerator (OFA), a field programmable gate array (FPGA), a neuromorphic chip, a quantum processing unit (QPU), an associative processing unit (APU), an arithmetic logic unit (ALU), an application-specific integrated circuit (ASIC), a floating-point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) element, etc.

[0215] The communication interface 1010 may include one or more receivers, transmitters, and / or transceivers enabling the computing device 1000 to communicate with other computing devices via an electronic communication network (including wired and / or wireless communication). The communication interface 1010 may include components and functions for enabling communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wirelessband), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 1020 and / or the communication interface 1010 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via interconnect system 1002 to one or more GPUs 1008 (e.g., memory of one or more GPUs 1008).

[0216] I / O port 1012 enables computing device 1000 to be logically coupled to other devices including I / O component 1014, one or more presentation components 1018, and / or other components, some of which may be built into (e.g., integrated into) computing device 1000. Illustrative I / O component 1014 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dish antennas, scanners, printers, wireless devices, etc. I / O component 1014 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by the user. In some cases, input may be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, pen recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of computing device 1000. Computing device 1000 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. Additionally, the computing device 1000 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables motion detection. In some examples, the computing device 1000 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0217] The power supply 1016 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 1016 may provide power to the computing device 1000 so that the components of the computing device 1000 can operate.

[0218] The presentation component 1018 can include a display (e.g., a monitor, a touch screen, a television screen, a heads-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 1018 can receive data from other components (e.g., the GPU 1008, the CPU 1006, etc.) and output the data (e.g., as images, video, sound, etc.).

[0219] Example data center

[0220] Figure 11 An example data center 1100 that can be used in at least one embodiment of the present disclosure is shown. The data center 1100 can include a data center infrastructure layer 1111, a framework layer 1120, a software layer 1130, and / or an application layer 1140.

[0221] As Figure 11 shown, the data center infrastructure layer 1111 can include a resource orchestrator 1112, grouped computing resources 1114, and node computing resources (“node C.R.s”) 1116(1)-1116(N), where “N” represents any whole, positive integer. In at least one embodiment, the node C.R.s 1116(1)-1116(N) can include, but are not limited to, any number of central processing units (“CPUs” or “processors”) (including accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic random access memory), storage devices (e.g., solid state or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules, and / or cooling modules, etc. In some embodiments, one or more of the node C.R.s 1116(1)-1116(N) can correspond to a server having one or more of the above-described computing resources. Moreover, in some embodiments, the node C.R.s 1116(1)-1116(N) can include one or more virtual components, such as a vGPU, a vCPU, etc., and / or one or more of the node C.R.s 1116(1)-1116(N) can correspond to a virtual machine (VM).

[0222] In at least one embodiment, grouped computing resources 1114 can include individual groupings of node C.R.s 1116 housed within one or more racks (not shown), or housed within a number of racks within data centers at different geographic locations (also not shown). Individual groupings of node C.R.s 1116 within grouped computing resources 1114 can include grouped computing, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, a number of node C.R.s 1116 including CPUs, GPUs, DPUs, and / or other processors can be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks can also include any number of power modules, cooling modules, and / or network switches in any combination.

[0223] Resource orchestrator 1122 can configure or otherwise control one or more node C.R.s 1116(1)-1116(N) and / or grouped computing resources 1114. In at least one embodiment, resource orchestrator 1122 can include a software design infrastructure (“SDI”) management entity for data center 1100. Resource orchestrator 1122 can include hardware, software, or some combination thereof.

[0224] In at least one embodiment, as Figure 11As shown, the framework layer 1120 can include a job scheduler 1128, a configuration manager 1134, a resource manager 1136, and / or a distributed file system 1138. The framework layer 1120 can include a framework that supports software layer 1130 and / or one or more applications 1142 of the application layer 1140. The software 1132 or the applications 1142 can contain network-based services software or applications, respectively, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1120 can be, but is not limited to, a type of free and open-source software web application framework (such as Apache Spark™ (hereinafter “Spark”)) that can utilize the distributed file system 1138 for large-scale data processing (e.g., “big data”). In at least one embodiment, the job scheduler 1128 can include a Spark driver to facilitate scheduling workloads supported by different layers of the data center 1100. The configuration manager 1134 can be capable of configuring different layers, such as the software layer 1130 and the framework layer 1120 (which includes Spark and the distributed file system 1138 for supporting large-scale data processing). The resource manager 1136 can be capable of managing clustered or grouped computing resources that are mapped to or allocated for supporting the distributed file system 1138 and the job scheduler 1128. In at least one embodiment, the clustered or grouped computing resources can include the grouped computing resources 1114 of the data center infrastructure layer 1111. The resource manager 1136 can coordinate with the resource coordinators 1112 to manage these mapped or allocated computing resources.

[0225] In at least one embodiment, the software 1132 included in the software layer 1130 can include software used by at least portions of the node C.R.s 1116(1)-1116(N), the grouped computing resources 1114, and / or the distributed file system 1138 of the framework layer 1120. One or more types of software can include, but are not limited to, internet web page search software, e-mail virus scanning software, database software, and streaming video content software.

[0226] In at least one embodiment, applications 1142 included in application layer 1140 can include one or more types of applications used by at least portions of node C.R.s 1116(1)-1116(N), grouped computing resources 1114, and / or distributed file system 1138 of framework layer 1120. One or more types of applications can include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0227] In at least one embodiment, any of configuration manager 1134, resource manager 1136, and resource orchestrator 1112 can implement any number and type of self-modification actions based on any quantity and type of data acquired in any technically feasible manner. Self-modification actions can free data center operators of data center 1100 from making potentially poor configuration decisions and can avoid underutilization and / or poor performance portions of data center.

[0228] According to one or more embodiments described herein, data center 1100 can include tools, services, software, or other resources to train one or more machine learning models or use one or more machine learning models to predict or infer information. For example, one or more machine learning models can be trained by computing weight parameters according to a neural network architecture using software and / or computing resources described above with respect to data center 1100. In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks can be used to infer or predict information using resources described above with respect to data center 1100 by using weight parameters computed through one or more training techniques such as, but not limited to, those described herein.

[0229] In at least one embodiment, data center 1100 can use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or virtual computing resources corresponding thereto) to perform training and / or inference using resources described above. Further, one or more software and / or hardware resources described above can be configured as a service that allows users to train or perform inference on information, such as image recognition, speech recognition, or other artificial intelligence services.

[0230] Example network environment

[0231] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 10 This is implemented on one or more instances of computing devices 900—for example, each device may include similar components, features, and / or functions of one or more computing devices 900. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of a data center 1100, examples of which are described in this document. Figure 11 To describe in more detail.

[0232] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0233] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein for the server can be implemented on any number of client devices.

[0234] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or application may respectively include network-based service software or applications. In embodiments, one or more client devices may use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open-source software network application framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0235] A cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions described herein (or one or more portions thereof). Any of these different functions can be distributed across multiple locations from a central or core server (e.g., can be distributed across one or more data centers of a state, region, country, globally, etc.). The core server can designate at least a portion of the functions to an edge server if the connection to the user (e.g., client device) is relatively close to the edge server. The cloud-based network environment can be private (e.g., limited to a single organization), can be public (e.g., available to many organizations), and / or combinations thereof (e.g., a hybrid cloud environment).

[0236] The one or more client devices can include at least some of the components, features, and functionality of one or more example computing devices 900 described herein with respect to FIG. 9. By way of example and not limitation, a client device can be implemented as a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smartwatch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a video camera, a surveillance device or system, a vehicle, a boat, a spaceship, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of these depicted devices, or any other suitable device.

[0237] The present disclosure can be described in the general context of computer code or machine-useable instructions, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules including routines, programs, objects, components, data structures, and the like, refer to code that performs particular tasks or implements particular abstract data types. The present disclosure can be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general- purpose computers, more specialty computing devices, and the like. The present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0238] As used herein, the term "and / or," with respect to a listing of two or more elements, means that at least one of the listed elements is present. For example, the phrase "A, B, and / or C" can include A alone, B alone, C alone, A and B, A and C, B and C, or A, B, and C. Also, the phrase "at least one of A or B" can include A alone, B alone, or A and B. Further, the phrase "at least one of A and B" can include A alone, B alone, or A and B.

[0239] The subject matter of the present disclosure is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the disclosure. Rather, the disclosure herein is intended to be illustrative of the subject matter of the present disclosure. The subject matter of the present disclosure, however, can be embodied in other ways than those specifically set forth herein without departing from the essence of the disclosure. Further, although the terms "step" and / or "block" can be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein commensurate with the purposes of the description herein.

[0240] Figure 12 FIGURE 12 illustrates a block diagram of one or more systems 1202 that can perform at least a portion of the processes described herein in accordance with some embodiments of the present disclosure. As shown, the one or more systems 1202 can include one or more processors 1204 (which can include and / or be similar to one or more CPUs 906, one or more GPUs 908, one or more CPUs 918, one or more GPUs 920, one or more CPUs 1006, or one or more GPUs 1008), one or more communication interfaces 1206 (which can include or be similar to network interface 924 or communication interface 1010), and memory 1208 (which can include and / or be similar to memory 1004). Additionally, memory 1004 can store one or more parsing components 102, schema information 104, one or more databases 108, one or more hint components 110, hint data 114, and one or more analysis components 118. Additionally, one or more processors 1204 can process one or more parsing components 102, one or more hint components 110, and / or one or more analysis components 118 to perform at least a portion of the processes described herein.

[0241] Example Paragraph

[0242] A. A method comprising: obtaining one or more software architecture documents associated with software and a hint associated with performing one or more validation tasks pertaining to the one or more software architecture documents; generating one or more input tokens associated with at least one or more portions of the one or more software architecture documents and the hint; generating, using one or more language models and based on at least the one or more input tokens, one or more output tokens representing information associated with the one or more validation tasks; and performing one or more operations based on at least the information.

[0243] B. The method of paragraph A, further comprising: generating one or more graphs associated with one or more relationships described in the one or more software architecture documents; and identifying, based on at least the one or more graphs, the one or more portions of the one or more software architecture documents that are relevant to the hint.

[0244] C. The method of paragraph B, wherein at least one of the one or more graphs represents at least: an indication of a requirement associated with the software; and the one or more relationships between the requirement and at least one of one or more functions, one or more visualizations, one or more descriptions, or text.

[0245] D. The method of paragraph B, wherein the identifying the one or more portions of the one or more software architecture documents that are relevant to the hint comprises: determining, using the one or more language models and based on at least input data representing the hint, that the hint includes one or more identifiers associated with content included in the one or more software architecture documents; and identifying, based on at least matching the one or more identifiers to at least a portion of the one or more graphs, the one or more portions of the one or more software architecture documents that are relevant to the hint.

[0246] E. The method of any of paragraphs A-D, wherein the one or more validation tasks are associated with determining whether an architecture of the software is correct; and the information comprises at least one of: first information indicating that the architecture is correct; second information indicating that the architecture includes one or more issues; or third information indicating one or more techniques for mitigating the one or more issues.

[0247] F. The method of any of paragraphs A-E, wherein the one or more validation tasks are associated with determining whether the software includes one or more vulnerabilities; and the information includes at least one of: first information indicating that the software does not include the one or more vulnerabilities; second information indicating that the architecture includes at least one vulnerability from the one or more vulnerabilities; or third information indicating one or more techniques for mitigating the vulnerabilities.

[0248] G. The method of any of paragraphs A-F, further comprising: storing data representing a plurality of cues associated with performing tasks associated with the one or more software architecture documents; and receiving one or more inputs indicating at least the cue from the plurality of cues, wherein obtaining the cue is based at least on the one or more inputs indicating the cue.

[0249] H. The method of any of paragraphs A-G, further comprising: receiving one or more inputs indicating one or more identifiers associated with content included in the one or more software architecture documents; and generating an updated cue by updating the cue to include at least the one or more identifiers, wherein the one or more tokens are generated for the updated cue and the one or more portions of the one or more software architecture documents.

[0250] I. The method of any of paragraphs A-H, wherein the performing the one or more operations includes at least one of: outputting content associated with the information; or updating the one or more software architecture documents based at least on the information.

[0251] J. A system comprising: one or more processors to: obtain architecture information associated with software; generate, using one or more language models and based at least on input data representing at least a portion of the architecture information and a cue associated with performing one or more tasks, output data representing information associated with the one or more tasks; and perform one or more operations based at least on the information.

[0252] K. The system of paragraph J, wherein the one or more processors are further to: generate one or more graphs associated with one or more relationships described in the architecture information; and identify, based at least on the one or more graphs, the at least a portion of the architecture information that is relevant to the cue.

[0253] L. The method of paragraph K, wherein at least one of the one or more graphs represents at least: an indication of a requirement associated with the software; and the one or more relationships between the requirement and at least one of one or more functions, one or more visualizations, one or more descriptions, or text.

[0254] M. The system of paragraph K, wherein at least a portion of the architectural information that is relevant to the prompt is identified at least by: using the one or more language models and determining, based at least on second input data representing the prompt, that the prompt includes one or more identifiers associated with content included in the architectural information; and identifying the at least a portion of the architectural information that is relevant to the prompt based at least on matching the one or more identifiers to at least a portion of the one or more graphs.

[0255] N. The system of any of paragraphs J-M, wherein: the one or more tasks are associated with performing one or more validation checks associated with the software; and the information comprises at least one of: first information indicating that the software satisfies the one or more validation checks; second information indicating that the software includes one or more issues associated with the one or more validation checks; or third information indicating one or more techniques for mitigating the one or more issues.

[0256] O. The system of any of paragraphs J-N, wherein the one or more processors are further to: store data representing a plurality of prompts associated with performing tasks associated with the architectural information; receive one or more inputs indicating at least the prompt from the plurality of prompts; and obtain the prompt based at least on the one or more inputs.

[0257] P. The system of any of paragraphs J-O, wherein the one or more processors are further to: receive one or more inputs indicating one or more identifiers associated with content included in the architectural information; and generate the prompt to include at least the one or more identifiers.

[0258] Q. The system of any of paragraphs J-O, wherein the one or more processors are further to: generate one or more first input tokens associated with the at least a portion of the architectural information and one or more second input tokens associated with the prompt, wherein: the input data represents the one or more first input tokens and the one or more second input tokens; and the output data represents one or more output tokens corresponding to the information.

[0259] R. The system of any of paragraphs J-Q, wherein the system is included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing optical transport simulation; a system for performing collaborative content creation for 3D assets; a system providing one or more cloud gaming applications; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more visual language models (VLMs); a system for performing operations using one or more multi-modal language models; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implementing one or more multi-modal language models; a system using or deploying one or more inference microservices; a system incorporating or deploying one or more machine learning models in services or microservices and OS-level virtualization packages (e.g., containers); a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0260] S. one or more processors comprising: processing circuitry to cause output of data representative of information associated with one or more software architecture documents corresponding to software, wherein the output data is generated based at least on processing, by one or more language models, input data representative of at least a portion of the one or more software architecture documents and one or more prompts associated with validating the one or more software architecture documents.

[0261] T. The one or more processors of paragraph S, wherein the one or more processors are included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing optical transport simulation; a system for performing collaborative content creation for 3D assets; a system providing one or more cloud gaming applications; a system for performing one or more deep learning operations; a system implemented using edge devices; a system implemented using robots; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more visual language models (VLMs); a system for performing operations using one or more multi-modal language models; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implementing one or more multi-modal language models; a system using or deploying one or more inference microservices; a system incorporating or deploying one or more machine learning models in services or microservices and OS-level virtualization packages (e.g., containers); a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Claims

1. A method, the method comprising: Obtain one or more software architecture documents associated with the software, and prompts related to performing one or more verification tasks concerning the one or more software architecture documents; Generate one or more input tokens associated with at least one or more portions of the one or more software architecture documents and the prompt; Use one or more language models and at least based on the one or more input tokens to generate one or more output tokens representing information associated with the one or more verification tasks; as well as At least one or more operations may be performed based on the information provided.

2. The method of claim 1, further comprising: Generate one or more diagrams associated with one or more relationships described in the one or more software architecture documents; as well as Identify, at least based on the one or more diagrams, the one or more portions of the one or more software architecture documents that are related to the prompt.

3. The method of claim 2, wherein at least one of the one or more figures represents at least: Indications of requirements associated with the software; and The requirement is related to at least one of one or more functions, one or more visualizations, one or more descriptions, or text.

4. The method of claim 2, wherein identifying the one or more portions of the one or more software architecture documents associated with the prompt comprises: The prompt is determined using one or more language models and based at least on input data representing the prompt, to include one or more identifiers associated with content included in one or more software architecture documents; as well as The one or more portions of the software architecture document associated with the prompt are identified at least based on matching the one or more identifiers with at least a portion of the one or more diagrams.

5. The method of claim 1, wherein The one or more verification tasks are associated with determining whether the architecture of the software is correct; and The information includes at least one of the following: First information indicating that the architecture is correct; Secondary information indicating that the architecture includes one or more issues; or Third information indicating one or more techniques for alleviating the one or more problems.

6. The method of claim 1, wherein The one or more verification tasks are associated with determining whether the software contains one or more weaknesses; and The information includes at least one of the following: First information indicating that the software does not include the one or more weaknesses; The architecture is indicated to include second information from at least one of the one or more weaknesses; or Third information indicating one or more techniques for mitigating the aforementioned weakness.

7. The method of claim 1, further comprising: The storage represents data representing multiple prompts associated with the execution of tasks related to the one or more software architecture documents; as well as Receive one or more inputs that indicate at least one of the prompts from the plurality of prompts. The prompt is obtained based on at least one or more inputs that indicate the prompt.

8. The method of claim 1, further comprising: Receive one or more inputs that indicate the association of one or more identifiers with the content included in the one or more software architecture documents; as well as An updated prompt is generated by updating the prompt to include at least one or more of the identifiers. The one or more tokens are generated in connection with the update prompt and the one or more sections of the one or more software architecture documents.

9. The method of claim 1, wherein performing the one or more operations comprises at least one of the following: Output the content associated with the information; or At least based on the information, update one or more software architecture documents.

10. A system comprising: One or more processors, said one or more processors being used for: Obtain architectural information associated with the software; Output data representing information associated with the one or more tasks is generated using one or more language models and based at least on input data representing at least a portion of the architecture information and prompts associated with performing one or more tasks; as well as At least one or more operations may be performed based on the information provided.

11. The system of claim 10, wherein the one or more processors are further configured to: Generate one or more graphs associated with one or more relationships described in the architecture information; and The at least portion of the architectural information related to the prompt is identified based on at least one or more of the graphs.

12. The method of claim 11, wherein at least one of the one or more figures represents at least: Indications of requirements associated with the software; and The requirement is related to at least one of one or more functions, one or more visualizations, one or more descriptions, or text.

13. The system of claim 11, wherein at least a portion of the architectural information relating to the prompt is identified at least in the following ways: Using the one or more language models and at least based on second input data representing the prompt, it is determined that the prompt includes one or more identifiers associated with content included in the architecture information; and The at least portion of the architecture information related to the prompt is identified at least by matching the one or more identifiers with at least a portion of the one or more graphs.

14. The system of claim 10, wherein: The one or more tasks are associated with performing one or more verification checks related to the software; and The information includes at least one of the following: First information instructing the software to satisfy one or more verification checks; The software is instructed to include second information related to one or more issues associated with the one or more verification checks; or Third information indicating one or more techniques for alleviating the one or more problems.

15. The system of claim 10, wherein the one or more processors are further configured to: The storage represents data representing multiple prompts associated with the execution of tasks related to the architecture information; Receive at least one or more inputs indicating at least one of the prompts from the plurality of prompts; and The prompt is obtained based on at least one or more of the inputs.

16. The system of claim 10, wherein the one or more processors are further configured to: Receive one or more inputs indicating one or more identifiers associated with the content included in the architecture information; and The prompt is generated to include at least one or more of the identifiers.

17. The system of claim 10, wherein the one or more processors are further configured to: Generate one or more first input tokens associated with at least a portion of the architecture information and one or more second input tokens associated with the prompt. in: The input data represents the one or more first input tokens and the one or more second input tokens; as well as The output data represents one or more output tokens corresponding to the information.

18. The system of claim 10, wherein the system is included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing optical transmission simulation; A system for performing collaborative content creation for 3D assets; A system that provides one or more cloud gaming applications; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models; A system for performing one or more conversational AI operations; A system for generating synthetic data; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system that implements one or more multimodal language models; A system that uses or deploys one or more inference microservices; A system that combines or deploys one or more machine learning models in a service or microservice and an OS-level virtualization package (e.g., a container); A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

19. One or more processors, said one or more processors comprising: A processing circuit system that outputs data representing information associated with one or more software architecture documents corresponding to software, wherein the output data is generated by processing input data based on at least one or more language models, the input data representing at least a portion of the one or more software architecture documents and one or more prompts associated with verifying the one or more software architecture documents.

20. The processor of claim 19 or more, wherein the processor is included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing optical transmission simulation; A system for performing collaborative content creation for 3D assets; A system that provides one or more cloud gaming applications; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models; A system for performing one or more conversational AI operations; A system for generating synthetic data; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system that implements one or more multimodal language models; A system that uses or deploys one or more inference microservices; A system that combines or deploys one or more machine learning models in a service or microservice and an OS-level virtualization package (e.g., a container); A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

Images