Software program test generation for computing systems and applications

Abstract

The invention relates to software program test generation for computing systems and applications. Embodiments of the present disclosure relate to applications, platforms, architectures, etc. for automated software demand verification. In particular, one or more Generative Language Model (GLM) cues may be generated based at least on program information describing a software program and based on demand information corresponding to demands of the software program. Based on these cues, the GLM may be capable of automatically identifying segments of software program information related to the demand. Furthermore, based on the identified segments and GLM cues, the GLM may be able to automatically create a test architecture (e.g., based on one or more additional cues) that may be used to verify whether the software program meets the demand.

Description

Background Technology

[0001] Software testing is typically a labor-intensive process. For example, a software program may have multiple different requirements related to its target behavior. Often, for a given software requirement or requirement verification step, engineers can manually search the software program's documentation ("program documentation") for corresponding sections that might relate to the target behavior. For instance, engineers might manually review the corresponding software architecture documentation and / or interface specification documentation to identify segments in such documentation that correspond to the target behavior. After identifying the software architecture and interface specification segments, engineers typically review a list of individual requirements and their corresponding requirement verification steps, and create multiple test cases for each requirement. These test cases can then be used as a benchmark for integration testing or unit testing, which can be code used to implement the test cases and verify that the software program meets the corresponding requirements. This process can be performed for each requirement, which is tedious and requires significant manpower. Summary of the Invention

[0002] Embodiments of this disclosure relate to automated software program test generation. Specifically, systems and methods for automating one or more aspects of software program testing are disclosed. For example, one or more generative language model (GLM) hints can be generated based at least on program information describing the software program. For example, program information may include the program itself, such as architecture documentation (generally referred to herein as "program architecture" or "architecture") and / or interface specification documentation (also referred to herein as "program interface specification" or "interface specification"). In these and other embodiments, GLM hints can be generated based on requirement information describing the requirements corresponding to the software program.

[0003] Based on such prompts, GLM can automatically identify requirement-related fragments in software program information. For example, GLM can automatically identify one or more requirement-related parts in the program architecture and / or one or more requirement-related parts in the program interface specification. Furthermore, based on the identified fragments and GLM prompts, GLM can automatically create (e.g., based on one or more additional prompts) a test architecture that can be used to verify whether the software program meets the requirements.

[0004] For example, in some embodiments, the test architecture may include a test specification that defines the behavior that can be used to verify whether a software program meets requirements. In these and other embodiments, the test specification may include one or more test cases that may indicate one or more verification steps that can be used to verify such behavior. Additionally or alternatively, the test architecture may include a test implementation for testing whether requirements are met. The test implementation may include code or routines that can be implemented in the programming language of the software program to verify the test specification. The test implementation can then be used to test the software program.

[0005] The embodiments described herein can provide significant improvements in the field of software testing and development. Specifically, automating the generation of test architectures for software programs can significantly reduce the amount of time required to test and deploy software programs. Furthermore, these embodiments contribute to improving the functionality of the computing system itself by providing a specific mechanism and a series of steps and operations that allow the computing system to automatically generate such test architectures. Attached Figure Description

[0006] The system and method for generating software program testing will now be described in detail with reference to the accompanying drawings, in which:

[0007] Figure 1 An example system relating to a test software program according to one or more embodiments of the present disclosure is shown;

[0008] Figure 2 An example process, which can be executed to generate a test architecture according to one or more embodiments of the present disclosure, is shown;

[0009] Figure 3 An example architecture fragment identification process according to one or more embodiments of this disclosure is illustrated;

[0010] Figure 4 An example program interface specification fragment identification process according to one or more embodiments of this disclosure is illustrated;

[0011] Figure 5 An example test specification generation process according to one or more embodiments of this disclosure is illustrated;

[0012] Figure 6 An example test implementation generation process according to one or more embodiments of this disclosure is illustrated;

[0013] Figure 7 This is a flowchart of a method for generating a test architecture for a software program according to one or more embodiments of the present disclosure;

[0014] Figure 8A This is a block diagram of an example generative language model system applicable to implementing at least some embodiments of the present disclosure;

[0015] Figure 8B According to one or more embodiments of this disclosure Figure 8A The generative LM includes a block diagram of an example implementation of a converter encoder-decoder scheme;

[0016] Figure 8C According to one or more embodiments of this disclosure Figure 8A The generative LM includes a block diagram of an example implementation of a decoder-converter architecture only;

[0017] Figure 9 This is a block diagram of an example computing device suitable for implementing one or more embodiments of the present disclosure;

[0018] Figure 10 This is a block diagram of an example data center applicable to implementing one or more embodiments of the present disclosure;

[0019] Figure 11A This is an illustration of an example autonomous vehicle according to one or more embodiments of the present disclosure;

[0020] Figure 11B According to one or more embodiments of this disclosure Figure 11A Examples of camera positions and fields of view for autonomous vehicles;

[0021] Figure 11C According to one or more embodiments of this disclosure Figure 11A A block diagram of an example system architecture for an example autonomous vehicle; and

[0022] Figure 11D It is according to one or more embodiments of this disclosure for use in cloud-based servers and Figure 11A Here is a system diagram illustrating communication between autonomous vehicles. Detailed Implementation

[0023] The systems and methods disclosed herein relate to the automation of software program testing. Specifically, this disclosure relates to creating a mechanism in which a generative language model (GLM) can be used to generate test architectures for a software program. For example, as discussed in detail in this disclosure, one or more embodiments can be configured to generate hints for a GLM, which uses these hints to generate test architectures.

[0024] In some embodiments, prompts may be generated that enable the GLM to identify which parts of the program information (e.g., program documentation describing various aspects of the program) may be related to the functionality of the software program corresponding to a specific requirement associated with the software program. For example, this requirement may include the target functionality of the software program. Additionally or alternatively, the requirement may include the target performance of the software program. In these and other embodiments, the requirement may include the target behavior of the software program in response to certain inputs. Additionally or alternatively, the requirement may include certain requirements related to software program testing, such as requirements related to the type of test to be performed.

[0025] The prompts also enable GLM to identify test specifications (as part of a test architecture) based on identified portions of program information. These test specifications define the behavior of a software program that indicates whether the program meets requirements. In these and other embodiments, GLM may be able to identify one or more test cases based on prompts (as part of a test specification). These test cases may indicate one or more operations (referred to herein as "verification steps") that the software program can perform, the results of which can help verify whether the software program's behavior is consistent with the behavior defined with respect to the test specification.

[0026] Additionally or alternatively, one or more embodiments of this disclosure may relate to using GLM for prompt-based test implementation generation. The test implementation may be included in a test architecture and may include code or routines that can be implemented in the programming language of the software program to achieve the test specification, such that executing the test implementation on the software program can be used to determine whether the software program performs the behavior defined by the test specification, thereby determining whether the software program meets the requirements.

[0027] Furthermore, these embodiments can be widely applied to various types of software program testing. For example, testing may include functional testing (used to determine whether a software program has a certain function), security testing (used to determine whether a software program has certain vulnerabilities) (including "fuzz testing," in which various abnormal inputs are provided to the software program), and so on. Additionally or alternatively, other types of testing may include unit testing, performance testing, and / or integration testing.

[0028] In these and other embodiments, the system and method can be implemented on a variety of different platforms, such as network security environments (e.g., LaunchPad), simulation environments (e.g., Drive ), software development kits (e.g., DriveWorks Omniverse), software application toolkits (e.g., of Toolkit) or any other suitable platform that can be used to develop software.

[0029] The systems and methods described herein can be used for a variety of purposes, such as, but not limited to, machine (e.g., robots, vehicles, construction machinery, warehouse vehicles / machines, autonomous, semi-autonomous and / or other machine types) control, machine motion, machine driving, synthetic data generation, model training (e.g., using real data, augmented data and / or synthetic data, such as synthetic data generated using simulation platforms or systems, synthetic data generated by synthetic data generation techniques (e.g., but not limited to the techniques described herein), perception, augmented reality (AR), virtual reality (VR), mixed reality (MR), robotics, security and supervision (e.g., in smart city implementations), autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or participant simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), distributed or collaborative content creation of 3D assets (e.g., using generic scene descriptor (USD) data, such as OpenUSD and / or other data types), cloud computing, generative artificial intelligence (e.g., using one or more diffusion models, converter models, etc.) and / or any other suitable application.

[0030] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots or robotic platforms, aviation systems, medical systems, marine systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations (e.g., in driving or vehicle simulations, in robot simulations, in smart city or supervised simulations, etc.), systems for performing digital twin operations (e.g., in conjunction with collaborative content creation platforms or systems, such as, but not limited to, NVIDIA's OMNIVERSE and / or other platforms, systems, or services using USD or OpenUSD data types), systems implemented using edge devices, systems containing one or more virtual machines (VMs), and systems for performing... Systems that perform synthetic data generation operations (e.g., using one or more neural rendering fields (NERF), Gaussian sputtering techniques, diffusion models, converter models, etc.), systems that are at least partially implemented in a data center, systems for performing conversational AI operations, systems that implement one or more language models (e.g., one or more large language models (LLM), one or more visual language models (VLM), one or more multimodal language models, etc.), systems for performing optical transport simulations, systems for performing collaborative content creation of 3D assets (e.g., using generic scene descriptor (USD) data, such as OpenUSD, computer-aided design (CAD) data, 2D and / or 3D graphics or design data, and / or other data types), systems that are at least partially implemented using cloud computing resources, and / or other types of systems.

[0031] Furthermore, one or more embodiments of this disclosure may relate to software program testing associated with self-machines and / or components of one or more self-machines, which may include any suitable machine or system capable of performing one or more autonomous or semi-autonomous operations. Exemplary self-machines may include, but are not limited to, vehicles (land, sea, space, and / or air), robots, robotic platforms, etc. For example, a self-machine computing application may include one or more applications executable by an autonomous or semi-autonomous vehicle, such as those related to… Figures 11A-11D The example autonomous vehicle 1100 described herein is alternatively referred to as "vehicle 1100" or "self-machine 1100". In this disclosure, references to "autonomous vehicle" or "semi-autonomous vehicle" can include any vehicle that can be configured to perform one or more autonomous or semi-autonomous navigation or driving operations. Therefore, such vehicles can also include vehicles that require an operator or in which an operator can also perform such operations.

[0032] In some examples, machine learning models can be packaged as microservices—such as inference microservices (e.g., NVIDIA NIM)—which may include containers (e.g., operating system (OS) level virtualization packages) that may include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model “engine.” For example, an inference microservice may include the container itself and the model (e.g., weights and biases). In some cases, such as when the machine learning model is small enough (e.g., with a small enough number of parameters), the model may be included within the container itself. In other examples (e.g., when the model is large), the model may be hosted / stored in the cloud (e.g., in a data center) and / or hosted on-premises and / or at the edge (e.g., on a local server or computing device, but outside the container). In these embodiments, the model may be accessed via one or more APIs (e.g., a REST API). Therefore, in some embodiments, the machine learning models described herein can be deployed as inference microservices to accelerate model deployment on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, an optimized inference engine (e.g., using standardized AI model deployments to build execution software, such as NVIDIA's Triton inference server, and / or one or more APIs for high-performance deep learning inference, which may include inference runtime and model optimizations that can provide low latency and high throughput for production applications such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). The machine learning models described herein, as well as acceleration infrastructure, may be included as part of the microservice, which can be deployed using a single command and / or orchestrated and automatically scaled on the acceleration infrastructure using a container orchestration system (e.g., reaching data center scale on a single device). Thus, an inference microservice may include machine learning models (e.g., models optimized for high-performance inference), inference runtime software for executing the machine learning models and providing output / response to inputs (e.g., user queries, prompts, etc.), and enterprise management software for providing health checks, identity, and / or other monitoring capabilities. In some embodiments, the inference microservice may include software for in-situ replacement and / or updating of the machine learning models. When replacing or updating, the software performing the replacement / update can maintain the user configurations of the inference runtime software and the enterprise management software.

[0033] Embodiments of this disclosure will be explained with reference to the accompanying drawings. It should be understood that the drawings are schematic and illustrative representations of such exemplary embodiments and are not limiting, nor are they necessarily drawn to scale. In the drawings, unless otherwise stated, features with the same number represent the same structure and function.

[0034] refer to Figure 1 , Figure 1 An example system 100 associated with a test software program according to one or more embodiments of this disclosure is shown. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or as alternatives to the arrangements and elements shown, and certain elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and location. The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. For example, in some embodiments, the systems and methods described herein may use one or more generative language models (e.g., such as...). Figures 8A-8C As described in [the document], one or more computing devices (e.g., such as [the following]). Figure 9 (as described in) and / or one or more data centers (e.g., as described in) Figure 10 This is achieved as described in [the document / document].

[0035] Generally, system 100 can be configured to use a generative language model (GLM) 110 to generate a test architecture 102 for a software program. The software program can include any type of software and / or software collections that can be configured to perform operations. In some embodiments, the software program can be considered a software system that can include multiple different types of software programs and / or software modules. For example, in some embodiments, the software program can include an operating system and / or one or more software programs running on the operating system.

[0036] GLM 110 can include any suitable model that can be configured to generate language, such as a Large Language Model (LLM), a Visual Language Model (VLM), or any other suitable model. In some embodiments, references... Figures 8A-8C The generative language model described can be an example of GLM 110.

[0037] In these and other embodiments, system 100 may include a test module 112. In some embodiments, test module 112 may include code and routines configured to perform operations described with respect to test module 112. Additionally or alternatively, test module 112 may be implemented using hardware including one or more processors, CPUs, graphics processing units (GPUs), data processing units (DPUs), parallel processing units (PPUs), microprocessors (e.g., for performing or controlling the execution of one or more operations), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), accelerators (e.g., deep learning accelerators (DLAs)), one or more programmable vision accelerators (PVAs) (which may include one or more vector processing units (VPUs), one or more direct memory access (DMA) systems, one or more pixel processing engines (PPEs), etc.), and / or other processor types. In these and other embodiments, test module 112 may be implemented using a combination of hardware and software. In this disclosure, operations described as being performed by test module 112 may include operations that test module 112 may perform on its own or cause to be performed by another device. In some embodiments, the test module 112 may use one or more generative language models (e.g., such as...) Figures 8A-8C As described in [the document], one or more computing devices (e.g., such as [the following]). Figure 9 (as described in) and / or one or more data centers (e.g., as described in) Figure 10 This is achieved as described in [the document / document].

[0038] Test module 112 can be configured to interact with GLM 110 to help generate test architecture 102. For example, test module 112 can be configured to generate one or more hints 114, which can be provided to GLM 110.

[0039] Typically, prompt 114 may include a set of instructions provided to GLM 110 that can guide GLM 110's response and / or be used by GLM 110 to generate text based on the content of prompt 114. Prompt 114 may include specific information, context, questions or tasks, commands, and / or examples that GLM 110 can use to generate relevant output. In these and other embodiments, some elements of prompt 114 may include context, instructions, questions or tasks, and / or examples.

[0040] Context may include background or setup information that provides the framework for the response. Commands may include instructions or directives on what the GLM110 should do, such as answering questions, generating text, or completing sentences. Questions / tasks may include queries or tasks for the model to process. Additionally, examples may include suggestions for expected output, such as the expected format and / or style of the output.

[0041] As discussed in further detail and within the context of this disclosure, in some embodiments, prompt 114 can be used to obtain information from GLM 110 for generating test architecture 102. For example, prompt 114 may include specific contextual information, instructions, questions, tasks, and / or examples related to one or more requirements corresponding to a software program, enabling GLM 110 to provide information corresponding to whether the test software program meets one or more requirements. Similarly, as discussed in further detail in this disclosure, test module 112 may be configured to interface with and interact with GLM 110 to generate test architecture 102. For example, Figure 2 A sample process that can be used to generate test architecture 102 is described.

[0042] Test architecture 102 may include a set of operations or instructions corresponding to determining whether a software program meets one or more associated requirements. For example, in some embodiments, test architecture 102 may include test specification 120, which defines the behavior that can verify whether the software program meets a specific requirement. In these and other embodiments, the test specification may include one or more test cases that may indicate one or more verification steps that can be used to verify such behavior. Additionally or alternatively, test architecture 102 may include a test implementation 122 for testing whether the requirements are met. Test implementation 122 may include code or routines that implement the test specification in the programming language of the software program.

[0043] In some embodiments, the test module 112 may acquire requirement information 104, program information 106, and / or one or more prompt templates 108. In these and other embodiments, the test module 112 may be configured to generate prompts 114 based on requirement information 104, program information 106, and / or prompt templates 108.

[0044] The requirement information 104 may include any suitable information that may indicate one or more requirements corresponding to a software program. For example, in some embodiments, the requirement information 104 may indicate certain target functionalities of the software program. Additionally or alternatively, the requirement information 104 may indicate one or more target performance metrics of the software program, such as response time and throughput. In these and other embodiments, the requirement information 104 may indicate target behaviors of the software program in response to certain inputs. Additionally or alternatively, the requirement information 104 may indicate certain requirements related to software program testing, such as requirements related to the type of test to be performed. For example, in some embodiments, the requirement information 104 may indicate requirements for reliability testing, fault tolerance testing, security testing, stress testing, and / or availability testing.

[0045] In some embodiments, the requirement information 104 may indicate the requirements by including specific information corresponding to one or more requirements. Additionally or alternatively, the requirement information 104 may include one or more indicators (e.g., codes, flags, pointers, etc.) that may indicate where the corresponding requirement can be obtained and / or used to find the corresponding requirement.

[0046] Test module 112 can be configured to receive requirement information 104 and identify and / or retrieve one or more requirements based on requirement information 104. As indicated, test module 112 can identify and / or retrieve requirements based on one or more requirements being directly included in requirement information 104 and / or by looking up and / or retrieving one or more requirements based on corresponding indicators included in requirement information 104.

[0047] In these and other embodiments, the requirement information 104 may include one or more fields that may indicate the corresponding characteristics of a requirement. For example, Table 1 shows some example fields that may be included in the requirement information 104. The examples in Table 1 are in an automated environment (e.g., within the scope of this disclosure). Figures 11A-11D The specific context of the software program corresponding to the self-machine described in Table 1. However, the examples in Table 1 are not intended to be limiting, and the requirement information 104 may have any number of fields different from those specifically shown. Furthermore, in some embodiments, information corresponding to one or more fields may be explicitly included in the requirement information 104. Additionally or alternatively, information corresponding to one or more fields may be obtained (e.g., received, accessed, determined, etc.) by looking up other information, which may not have been initially included in the requirement information 104 but may be added to the requirement information 104 based on the requirement information 104 initially provided to the test module 112 (e.g., added by the test module 112).

[0048]

[0049]

[0050] The requirement information 104 can be configured in any suitable format. For example, in some embodiments, the requirement information 104 can be represented in JSON (JavaScript Object Notation) format as an object with key-value pairs. Each key can correspond to a specific attribute of the requirement, while the value can provide detailed information. For example, the information corresponding to the “Fields” column in Table 1 can include keys, while the information corresponding to the “Field Description” column can correspond to values. Additionally or alternatively, the requirement information 104 can be represented in HTML (Hypertext Markup Language) format or any other suitable format.

[0051] In these and other embodiments, the requirement information 104 may include metadata corresponding to substantive information about the requirements included in the requirement information 104. For example, the requirement information 104 may indicate the configuration format of the substantive requirement information 104 (e.g., JSON, HTML, etc.).

[0052] Program information 106 may include information that indicates one or more aspects of the software program. For example, in some embodiments, program information 106 may include documentation describing one or more aspects of the software program. In these and other embodiments, as discussed in further detail in this disclosure (e.g., regarding...), Figure 2 The program information 106 can be used to identify which parts of the software program correspond to specific requirements, and these parts can be used to generate a test architecture corresponding to the specific requirements.

[0053] In some embodiments, program information 106 may include software program architecture documentation 116 (“program architecture 116”). Program architecture 116 may include a collection of documents that describe in detail the architecture of the software program.

[0054] For example, program architecture 116 may include a high-level introduction to the software program, outlining its purpose, scope, and context. This introduction may include the goals the software program aims to achieve, such as performance, scalability, security, and / or maintainability.

[0055] In these and other embodiments, program architecture 116 may include an architecture description. This architecture description may describe the architectural styles and patterns employed in the software program, such as microservices or a layered architecture. In these and other embodiments, the architecture description may provide a high-level architecture diagram that captures the overall structure of the software program, its main components, and / or their interactions.

[0056] Additionally or alternatively, program architecture 116 may describe the components and modules of the software program, explaining the responsibilities, interfaces, and interactions of each component. In these and other embodiments, program architecture 116 may include a module view that decomposes the software program into modules or packages, illustrating the dependencies and relationships between them.

[0057] In these and other embodiments, program architecture 116 may include a data architecture corresponding to the software program. This data architecture may include data models, such as entity-relationship diagrams (ERDs) and class diagrams, and a description of the database architecture, such as schemas, tables, and relationships.

[0058] Additionally or alternatively, program architecture 116 may include a deployment architecture corresponding to the software program. The deployment architecture can describe the deployment environment by depicting a deployment diagram of hardware, network topology, and deployment nodes. In these and other embodiments, the deployment architecture may also include detailed information about setting up and configuring different environments (e.g., development, testing, and production).

[0059] In these and other embodiments, program architecture 116 may include indications of the interaction and behavior of the software program. For example, program architecture 116 may include use cases and scenarios that the software program can support. Additionally or alternatively, use cases and scenarios may be supplemented by sequence diagrams (which illustrate the interactions between components in various scenarios) and state diagrams (which illustrate the states and transitions of components or subsystems).

[0060] Additionally or alternatively, program architecture 116 may include technology stack sections that list the technologies, frameworks, libraries, and / or tools used in the software program. In these and other embodiments, program architecture 116 may include indications related to design decisions and trade-offs, which may be indicated via a decision log that records architectural decisions, their rationale and implications, and a description of the trade-offs made during the design process.

[0061] In these and other embodiments, program architecture 116 may outline quality attributes and / or testing strategies to help ensure that attributes such as performance, security, maintainability, and availability are met. This section may also describe system testing methodologies, including unit testing, integration testing, system testing, and performance testing. In some embodiments, such portions of program architecture 116 may be referenced in and / or included as part of requirements information 104.

[0062] Additionally or alternatively, in some embodiments, program architecture 116 may include a glossary and / or a reference section, the glossary providing definitions of key terms and concepts used in the documentation, and the reference section listing references such as standards, guidelines, and external documents.

[0063] In some embodiments, program information 106 may also include a program interface specification 118 (“Interface Specification 118”) corresponding to the software program. Typically, Interface Specification 118 provides guidance on how to integrate and use the interface corresponding to the software program to help ensure consistency and compatibility throughout the software program. Overall, Interface Specification 118 serves as a reference document that can describe or be used to identify communication between different parts of the software program and between the software program and external entities.

[0064] For example, in some embodiments, interface specification 118 may include a description of how different components of the software program interact with each other and how different components of the software program interact with external systems. For example, interface specification 118 may define various interfaces, including APIs (Application Programming Interfaces), user interfaces, and communication protocols that the software program can use. Additionally or alternatively, interface specification 118 may specify methods, parameters, data formats, and / or return types corresponding to the interfaces.

[0065] In these and other embodiments, interface specification 118 may outline the expected behavior of each interface, such as error handling, performance requirements, and security constraints. Additionally or alternatively, interface specification 118 may include examples and use cases to illustrate typical interactions between interfaces.

[0066] In these and other embodiments, program information 106 may include metadata corresponding to substantive information about the program documentation included in program information 106. For example, program information 106 may indicate the configuration format of the substantive program information (e.g., JSON, HTML, etc.).

[0067] The prompt template 108 may include a framework that the test module 112 can use to generate one or more prompts 114. For example, in some embodiments, the prompt template 108 may include pre-filled language that provides context, poses a question, describes a task, describes a command, and / or describes examples that may be included in the prompts 114.

[0068] In some embodiments, the test module 112 may include a generative language model and may be configured to generate at least a portion of the pre-filled language included in the prompt template 108. Additionally or alternatively, at least a portion of the pre-filled language may be manually entered.

[0069] In these and other embodiments, the prompt template 108 may include one or more input fields that can be populated with specific information, such that a corresponding pre-filled language can be tailored to a specific context, question, task, command, and / or example based on the information included in the field. For example, in some embodiments, one or more fields may be configured to be populated with at least a portion of the requirement information 104. Therefore, the pre-filled language corresponding to such fields may be modified to correspond to a specific requirement associated with the specific requirement information 104 used to populate such fields. In this disclosure, references to populating a field with "requirement information 104" may include populating the field with any information about a specific requirement that can be obtained from the requirement information 104.

[0070] In some embodiments, the prompt template 108 may be a code block executable by the test module 112. For example, the prompt template 108 may include instructions executable by the test module 112, which may receive certain inputs and then generate corresponding prompts based on these inputs and their corresponding instructions.

[0071] Additionally or alternatively, as further described in this disclosure (e.g., as regarding...). Figure 2 As discussed, one or more fields can be configured to be populated with program information 106. In this disclosure, mentioning populating a field with "program information 106" can include populating the field with any information about program information 106 and / or the software program itself that can be obtained from program information 106.

[0072] In these and other embodiments, as discussed in further detail in this disclosure (e.g., regarding...), Figure 2 As discussed herein, one or more fields may be configured to be populated with additional information available to the test module 112 during the generation of the test architecture 102. For example, in some embodiments, the additional information may include information output by the GLM 110 in response to receiving one or more prompts 114, as discussed in further detail herein.

[0073] In some embodiments, system 100 may include a storage library 124. Storage library 124 may include any suitable computer-readable storage medium that can be used to store information. In some embodiments, storage library 124 may operate as a local cache corresponding to test module 112. In these and other embodiments, storage library 124 may be configured to store information related to the generation of the test architecture. For example, storage library 124 may be configured to store information that can be obtained from the output of GLM 110 based on prompt 114. As discussed in further detail (e.g., regarding...) Figure 2As discussed, test module 112 can be configured to determine whether information stored in repository 124 is available for generating subsequent test architectures 102 for one or more other requirements. In response to determining that the stored information is available, test module 112 can retrieve such information from repository 124, rather than generating corresponding prompts and retrieving such information from GLM 110 based on those prompts. Therefore, repository 124 can, in certain circumstances, help improve the efficiency of generating certain test architectures.

[0074] Therefore, system 100 can be configured to generate test architecture 102 based on interaction with GLM 110. Modifications, additions, or deletions can be made to system 100 without departing from the scope of this disclosure. In these and other embodiments, prompt template 108 can be omitted and / or significantly simplified, wherein the generative language model of test module 112 can have simple instructions to analyze requirement information 104 and program information 106 and generate test architecture 102 accordingly. In these and other embodiments, in some embodiments, system 100 can be configured to perform actions such as those related to generating test architecture 102 when generating test architecture 102. Figure 2 The one or more operations discussed. Furthermore, although... Figure 1 While other parts of this disclosure are described with respect to test software programs, the principles and techniques discussed can be used to generate test architectures for any suitable computing system, which may include hardware, software, and / or a combination of hardware and software.

[0075] Figure 2 Example processes 200 executable to generate a test architecture according to one or more embodiments of this disclosure are illustrated. Each operation or block of process 200 described herein may include a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. Process 200 may also be embodied as computer-usable instructions stored on a computer storage medium. Process 200 may be provided by a standalone application, service, or managed service (independently or in combination with other managed services), or as a microservice via an application programming interface (API), or as a plug-in to other products, to name a few. Furthermore, as an example, regarding Figure 1The system described herein describes process 200. However, process 200 may also be performed additionally or alternatively by any single system or combination of systems, including but not limited to the systems described herein. Furthermore, for ease of explanation, the description of process 200 pertains to generating a test architecture corresponding to the “requirements” associated with software program 250, but this process can also be used to generate test architectures for any number of requirements for any number of software programs. Software program 250 may include any suitable software program, such as any software program described in this disclosure.

[0076] Process 200 may include data collection operation 202 (“Data Collection 202”). Data Collection 202 may include aggregating information (e.g., from...) Figure 1 The test module 112 in the document (which contains the information) can be used to generate a test architecture corresponding to the requirements. For example, data collection 202 may include acquiring (e.g., receiving, accessing, reading, etc.) requirement information 204. In some embodiments, Figure 1 The requirement information 104 in the data collection process can be an example of requirement information 204. In these and other embodiments, data collection 202 may include acquiring (e.g., receiving, accessing, reading, etc.) program information 206 corresponding to software program 250. In some embodiments, Figure 1 The program information 106 in the example can be an example of program information 206. In these and other embodiments, program information 206 may include program architecture 216 (which may be related to...). Figure 1 The program architecture 116 is similar to or analogous to the program interface specification 218 (which may be similar to or analogous to the program architecture 116) and / or the program interface specification 218 (which may be similar to or analogous to the program interface specification 218). Figure 1 (Similar to or similar to the program interface specification 118 in the document).

[0077] In some embodiments, data collection 202 may include verifying whether all the general input information required to at least begin generating a test architecture is available. For example, it may be determined that requirement information 204 includes information corresponding to identified requirements for which the software program 250 will be tested. For example, it may be determined whether requirement information 204 includes identifiers and / or descriptive text corresponding to the identified requirements. Additionally or alternatively, if the requirement information includes identifiers for finding a specific requirement, it may be determined whether the requirement can be obtained using the provided identifiers. Additionally or alternatively, it may be determined that program information 206 is complete enough to identify which parts of the software program 250 correspond to the identified requirements.

[0078] In these and other embodiments, data collection 202 may include normalizing the acquired information into a common format (e.g., plain text, JSON format, etc.). For example, requirement information 204 and program information 206 may be analyzed to determine whether they are in a specified format. Parts that are not in a specified format may be converted to such a format.

[0079] In some embodiments, process 200 may include a segment identification operation 208 (“segment identification 208”). Segment identification 208 may include identifying potentially requirement-corresponding portions of program information 206 (also referred to as “segments”) as one or more information segments 230. For example, in some embodiments, segment identification 208 may include identifying one or more segments of program architecture 216 and / or one or more segments of interface specification 218 that may correspond to requirements.

[0080] In some embodiments, fragment identification 208 may include architecture fragment identification operation 210 (“architecture fragment identification 210”), configured to identify one or more architecture fragments 232 that may be included in information fragment 230. These architecture fragments 232 may include fragments of program architecture 216 corresponding to requirements. In this disclosure, references to “architecture fragment 232” may also refer to information about where the architecture fragment 232 can be accessed; therefore, the term “architecture fragment 232” does not necessarily refer to an actual architecture fragment, but may also refer to information about architecture fragments. In some embodiments, architecture fragment identification 210 may include information about this disclosure. Figure 3 Describe one or more operations.

[0081] Additionally or alternatively, fragment identification 208 may include interface fragment identification operation 212 (“interface fragment identification 212”), which is configured to identify one or more interface fragments 234 that may be included in information fragment 230. Interface fragment 234 may include fragments of interface specification 218 corresponding to requirements. In this disclosure, references to “interface fragment 234” may also refer to information about where interface fragment 234 can be accessed; therefore, the term “interface fragment 234” does not necessarily refer to an actual interface fragment, but may also refer to information about interface fragments. In some embodiments, interface specification fragment identification 212 may include information about this disclosure. Figure 4 Describe one or more operations.

[0082] In some embodiments, fragment identification 208 may include generating one or more fragment identification prompts, which may be provided to the GLM, for example... Figure 1 The test module 112 can generate one or more fragment recognition prompts, which can be provided to Figure 1GLM 110 in this embodiment and other embodiments. In these embodiments, fragment recognition prompts may be generated based on requirement information 204, program information 206, and one or more program information prompt templates 224.

[0083] In some embodiments, the program information prompt template 224 may be included in Figure 1 In the prompt template 108. In these and other embodiments, the program information prompt template 224 can be specifically used to generate prompts that can be used to obtain one or more program information fragments 230 (e.g., one or more program architecture fragments 232) and / or one or more interface specification fragments 234, which may correspond to the requirements indicated by the requirement information 204.

[0084] For example, the program information prompt template 224 may include a pre-filled language for instructing the GLM to search for various portions of the program information 206. In these and other embodiments, the program information prompt template 224 may include input fields associated with the pre-filled language, which may be configured to be filled with at least some of the requirement information 204. In these and other embodiments, the program information prompt template 224 may include fields associated with the program information 206, which may be configured to be filled with at least some of the program information 206. The pre-filled language, combined with the filled fields corresponding to the requirement information 204 and the program information 206, can correspondingly generate a prompt instructing the GLM to search for portions of the program information 206 (as indicated by the filled program information fields) that may correspond to the requirement (as indicated by the filled requirement information field).

[0085] In some embodiments, the program information template 224 may include an architecture hint template 226. The architecture hint template 226 may be configured to instruct the GLM to search within the program architecture 216 for a fragment corresponding to the requirement associated with the requirement information 204. For example, the architecture hint template 226 may be configured to receive information about the program architecture 216 (e.g., as indicated by the program information 206) and portions of the requirement information 204 indicating the requirement and its corresponding format as input. The architecture hint template 226 may be configured to populate various fields with such information and may include a pre-populated language for interacting with such fields. The result may include generating an architecture hint configured to cause the GLM to search within the program architecture 216 for a fragment (e.g., architecture fragment 232) corresponding to the requirement associated with the requirement information 204.

[0086] For example, in some embodiments, the pre-filled language of the architecture hint template 226 may include general instructions for the GLM describing the GLM's task as analyzing program architecture 216 (as indicated by the provided inputs of program architecture 216) to obtain information related to requirements (as indicated by the provided inputs of requirement information 204). In these and other embodiments, the pre-filled language may indicate that the goal of analyzing program architecture 216 is to extract portions of program architecture 216 corresponding to requirements. Additionally or alternatively, the pre-filled language may indicate what uses the portions of program architecture 216 can be used for so that the GLM can better identify the relevant portions. In these and other embodiments, the pre-filled language may interpret certain characteristics corresponding to program architecture 216, such as the format of program architecture 216, the segments of program architecture 216, the nomenclature that program architecture 216 can use, etc. Additionally or alternatively, in some embodiments, the pre-filled language may include instructions related to certain types of information (e.g., comments, inline tags, etc.) that may be included in program architecture 216, which can be used to identify which portions may correspond to requirements.

[0087] In these and other embodiments, the pre-filled language may include additional instructions related to the current task, such as one or more directed instructions that describe the current task in more detail. Additionally or alternatively, the pre-filled language may describe how the GLM identifies the architectural fragment 232 corresponding to the requirement. For example, the pre-filled language may include one or more "do's" and / or "don't's" lists related to identifying the architectural fragment 232. In these and other embodiments, the pre-filled language may indicate to the GLM the format of the available output. Additionally or alternatively, the pre-filled language may include examples corresponding to the output, such as example output itself, examples of the output's appearance, examples of how the output is formatted, and so on.

[0088] Such as about Figure 3 In further detail, in some embodiments, fragment identification 208 may accordingly include providing applicable requirements information 204 and information about the program architecture 216 to the architecture hint template 226, and executing the architecture hint template 226 to generate a corresponding architecture hint. Fragment identification 208 may then include providing the architecture hint to the GLM, and the GLM may identify the architecture fragment 232 based on the architecture hint.

[0089] Additionally or alternatively, in some embodiments, the program information template 224 may include an interface hint template 228. The interface hint template 228 may be configured to instruct the GLM to search in the interface specification 218 for a fragment corresponding to the requirement associated with the requirement information 204. For example, the interface hint template 228 may be configured to receive information about the interface specification 218 (e.g., as indicated by program information 206) and portions of the requirement information 204 indicating the requirement and its corresponding format as input. The interface hint template 228 may be configured to populate various fields with such information and may include a pre-populated language for interacting with these fields. The result may include generating an interface specification hint configured to cause the GLM to search in the interface specification 218 for a fragment corresponding to the requirement associated with the requirement information 204.

[0090] For example, in some embodiments, the pre-filled language of the interface prompt template 228 may include general instructions for the GLM, which describe the GLM's task as analyzing the interface specification 218 (as indicated by the input instructions provided in the interface specification 218) to obtain information related to the requirements (as indicated by the input instructions provided in the requirement information 204). In these and other embodiments, the pre-filled language may indicate that the goal of analyzing the interface specification 218 is to extract the portions of the interface specification 218 corresponding to the requirements. Additionally or alternatively, the pre-filled language may indicate what uses the portions of the interface specification 218 can be used for, so that the GLM can better identify the relevant portions. In these and other embodiments, the pre-filled language may interpret certain characteristics corresponding to the interface specification 218, such as the format of the interface specification 218, the segments of the interface specification 218, the naming conventions that the interface specification 218 can use, etc. Additionally or alternatively, in some embodiments, the pre-filled language may include instructions related to certain types of information (e.g., comments, inline tags, etc.) that may be included in the interface specification 218, which can be used to identify which portions may correspond to the requirements.

[0091] In these and other embodiments, the pre-filled language may include additional instructions related to the current task, such as one or more directional instructions that describe the current task in more detail. Additionally or alternatively, the pre-filled language may describe how the GLM identifies the interface fragment 234 corresponding to the requirement. For example, the pre-filled language may include one or more "dos" and / or "don'ts" lists related to identifying the interface fragment 234. In these and other embodiments, the pre-filled language may indicate to the GLM the format of the available output. Additionally or alternatively, the pre-filled language may include examples corresponding to the output, such as example output itself, examples of the output's appearance, examples of how the output is formatted, etc.

[0092] Such as combination Figure 4 In further detail, in some embodiments, fragment recognition 208 may accordingly include providing applicable requirements information 204 and program information 206 to the interface specification hint template 228, and executing the interface specification hint template 228 to generate a corresponding interface specification hint. Fragment recognition 208 may then include providing the interface specification hint to the GLM, and the GLM may identify the interface specification fragment 234 based on the interface specification hint.

[0093] In some embodiments, process 200 may include test architecture generation operation 236 (“Test Architecture Generation 236”). Typically, test architecture generation 236 may be configured to generate one or more elements of a test architecture, such as those related to... Figure 1 The elements described in test architecture 102. Additionally or alternatively, in some embodiments, test architecture generation 236 may be configured to generate one or more portions of the test architecture based on one or more information fragments 230 (e.g., one or more architecture fragments 232 and / or one or more interface fragments 234), as discussed in further detail herein.

[0094] In these and other embodiments, test architecture generation 236 can be configured to generate one or more portions of a test architecture based on one or more test architecture hint templates 238. In some embodiments, the architecture hint template 238 may include... Figure 1 In the prompt template 108. In these and other embodiments, the architecture prompt template 238 can be specifically used to generate prompts that can be used to obtain one or more elements of a test architecture, as described in further detail herein.

[0095] In some embodiments, test architecture generation 236 may include a test specification generation operation 244 (“Test Specification Generation 244”), which is configured to generate test specification 220. Test specification 220 may be related to... Figure 1 The described test specification 120 is similar or analogous. In some embodiments, test specification generation 244 can be configured to generate test specification 220 based on information fragment 232, requirement information 204, and test specification prompt template 240 that may be included in test architecture prompt template 238.

[0096] For example, the test specification prompt template 240 can be configured to receive at least a portion of information fragment 232 and requirement information 204 as input. In these and other embodiments, the test specification prompt template 240 may include a pre-filled language for instructing the GLM to generate a test specification 220 based on the provided input.

[0097] For example, the pre-filled language may include general instructions that instruct the GLM to generate a set of test cases for a given requirement indicated by the requirement information 204 provided as input. In these and other embodiments, the pre-filled language may include instructions related to the format of the test cases.

[0098] Additionally or alternatively, in some embodiments, the pre-filled language may include more specific instructions that direct the GLM on how to generate test cases. For example, the pre-filled language may instruct the GLM to parse information fragment 232 provided as input to extract information about the software program 250 from the information fragment 232, which can be used to generate test cases corresponding to a given requirement. Additionally or alternatively, the pre-filled language may include instructions on how the GLM should begin generating test cases, and / or one or more "dos" and / or "don'ts" lists related to generating test specification 220.

[0099] In these and other embodiments, the pre-filled language may include certain requirements that test cases should meet and / or specific characteristics of test cases. Additionally or alternatively, the pre-filled language may include sample test cases that GLM can use to generate test cases.

[0100] Such as about Figure 5 In further detail, in some embodiments, test specification generation 244 may accordingly include providing applicable requirement information 204 and information fragment 230 to test specification hint template 240, and executing test specification hint template 240 to generate a corresponding test specification hint. Test specification generation 244 may then include providing the test specification hint to GLM, and GLM may generate test specification 220 based on the test specification hint.

[0101] In some embodiments, test architecture generation 236 may include a test implementation generation operation 246 (“test implementation generation 246”), which is configured to generate a test implementation 222. The test implementation 222 may be related to... Figure 1 The described test implementation method 122 is similar or analogous. In some embodiments, the test implementation method generation 246 can be configured to generate the test implementation method 222 based on the test specification 220, the information fragment 230, and the test implementation method prompt template 242 that may be included in the test architecture prompt template 238.

[0102] For example, the test implementation hint template 242 can be configured to receive information fragment 232 and test specification 220 (e.g., one or more test cases included in test specification 220) as input. In these and other embodiments, the test implementation hint template 242 may include a pre-filled language that instructs the GLM to generate a test implementation 222 based on the provided input.

[0103] For example, the pre-filled language may include generic instructions that instruct the GLM to generate code in a specific programming language (e.g., C code) that implements the test cases of test specification 220. In these and other embodiments, the pre-filled language may include instructions related to instructing the GLM to use information included in information fragment 232 in code generation.

[0104] Additionally or alternatively, in some embodiments, the pre-filled language may include more specific instructions instructing the GLM on how to generate test cases. For example, the pre-filled language may include certain requirements and / or specific characteristics of the code that the test implementation 222 should meet. Additionally or alternatively, the pre-filled language may include sample test cases that the GLM can use when generating test cases. Additionally or alternatively, the pre-filled language may include instructions on how the GLM should begin generating code and / or one or more "dos" and / or "don'ts" lists related to the generation of test implementation 222. In these or other embodiments, the pre-filled language may include sample code that the GLM can use in the generation of test implementation 222.

[0105] Such as about Figure 6 In further detail, in some embodiments, test implementation generation 246 may include providing test specification 220 and information fragment 230 to test implementation hint template 242, and executing test implementation hint template 242 to generate corresponding test implementation hints. Then, test implementation generation 246 may include providing the test implementation hints to GLM, and GLM may generate test implementation 222 based on the test implementation hints.

[0106] In some embodiments, process 200 may include software program testing operation 252 (“program test 252”). Typically, program test 252 may be configured to test software program 250 based on test implementation 222. For example, software program test 252 may include... Figure 1 The test module 112 executes the code of test implementation 222 for the software program 250 to run the test cases included in the test specification 220.

[0107] In these and other embodiments, program testing 252 may include evaluating the performance of tests on software program 250. For example, the coverage of software program 250 during the execution of test cases in test implementation 222 may be determined. For example, one or more suitable test coverage techniques may be used to gather information indicating which functionalities of software program 250 are covered by the execution of test implementation 222. Additionally or alternatively, it may be determined which lines of code, the percentage of lines of code, etc., may have been covered during the execution of test implementation 222.

[0108] In these and other embodiments, it can be determined whether program test 252 covers a threshold amount of software program 250. For example, it can be determined whether code of a threshold percentage of software program 250 has been executed and / or whether a threshold percentage of functionality has been tested. The threshold amount may vary due to certain test tolerances, specifications, requirements, etc. In some embodiments, the threshold amount may be based on heuristic analysis related to coverage and the satisfaction of such tolerances, specifications, requirements, etc.

[0109] In some embodiments, in response to determining that a threshold coverage amount is not met, program test 252 may be configured to update one or more prompts generated during process 200. For example, program test 252 may modify the prompts to target gaps in test coverage that can be identified from the identified coverage results. In these and other embodiments, updating the prompts may result in modifications to test implementation 222 (e.g., generating additional code for test implementation 222), thereby allowing for performance improvements to test implementation 222 based on the performance of software program test 252.

[0110] Therefore, process 200 can be configured to utilize one or more GLMs as part of the test software program 250. Using GLMs to generate test specifications 220 and test implementations 222 as part of process 200 can improve the overall efficiency of software testing, thereby improving the software development technology field.

[0111] Without departing from the scope of this disclosure, process 200 may be modified, added to, or deleted. For example, although the individual blocks or operations of process 200 are shown as discrete blocks or operations, they may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the specific implementation. Furthermore, in some embodiments, one or more operations may be combined into fewer operations, or expanded to include more operations.

[0112] For example, the number of prompts generated may vary depending on the implementation. For example, in some embodiments, two or more prompts may be combined into a single prompt. Additionally or alternatively, one or more prompts may be divided into additional prompts.

[0113] Furthermore, different operations can be performed by various elements that are different from those described above. For example, in some embodiments, it is described as being performed by a test module (e.g., Figure 1 One or more operations performed by the test module 112 in the test module may be performed by a GLM, which may or may not be included as part of the test module. Furthermore, in some embodiments, the same GLM may be used to perform all operations described with respect to that GLM. Additionally or alternatively, two or more GLMs may be used.

[0114] Figure 3 An example architectural fragment identification process 300 (“Process 300”) according to one or more embodiments of this disclosure is illustrated. Each operation or block of Process 300 described herein may include a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. Process 300 may also be embodied as computer-usable instructions stored on a computer storage medium. Process 300 may be provided by a standalone application, service, or managed service (independently or in combination with other managed services), or as a microservice via an application programming interface (API), or as a plug-in to another product, to name a few. For example, Process 300 may be provided by… Figure 1 The system Figure 9 computing devices and / or Figure 10 The process 300 is executed by one or more components in the data center. However, process 300 may be additionally or alternatively executed by any system or combination of systems, including but not limited to the systems described herein. Furthermore, for ease of interpretation, the description of process 300 is concerned with identifying architectural fragments related to “requirements” associated with software programs; however, such a process can be used to identify architectural fragments for any number of requirements for any number of software programs. The software program may include any suitable software program, such as any software program described in this disclosure.

[0115] Furthermore, in some embodiments, one or more operations of process 300 may be performed to perform actions related to... Figure 2 The described architecture fragment identification 210. However, process 300 is not limited to the implementation of architecture fragment identification 210.

[0116] Process 300 may include an architecture hint generation operation 302 (“Architecture Hint Generation 302”), which can be used to generate architecture hint 306. Architecture hint 306 may be generated based on requirements information 304, program architecture 316, and architecture hint template 326. Requirements information 304 may be related to… Figure 2The requirement information 204 is similar or related, and the program architecture 316 can be similar to... Figure 2 The program architecture 216 is similar to or similar to the architecture hint template 326. Figure 2 The architecture hint template 226 is similar or analogous.

[0117] In some embodiments, architecture suggestion generation 302 may include providing requirements information 304 and program architecture 316 as input (e.g., by...). Figure 1 The test module 112 provides the architecture hint template 326. In this disclosure, a reference to “program architecture 316” provided as input can refer to a program architecture that can be used to identify where program architecture 316 is found and / or accessed.

[0118] In these and other embodiments, architecture hint generation 302 may include executing architecture hint template 326 to generate architecture hint 306 based on the pre-filled language of architecture hint template 326 and the input requirements information and program architecture 316.

[0119] In these and other embodiments, process 300 may include GLM interaction operation 308 (“GLM interaction 308”). GLM interaction 308 may include providing architecture hints 306 to GLM 310, which can interact with… Figure 1 The GLM 310 is similar to or analogous to the GLM 110 in these and other embodiments. In these embodiments, the GLM 310 can identify one or more fragments of the program architecture 316 corresponding to the requirements associated with the requirement information 304 as architecture fragments 332. In some embodiments, the GLM 310 can output architecture fragments 332. Architecture fragments 332 can be associated with... Figure 2 The architecture fragment 232 is similar or analogous to it.

[0120] Additionally or alternatively, in addition to and / or as a supplement to architecture hint generation 302 and GLM interaction 308, process 300 may include a repository query operation 312 (“Repository Query 312”). Repository Query 312 may include accessing repository 324. Repository 324 may be connected to… Figure 1 The repository 124 is similar to or analogous to the repository 324 and may include one or more lists of one or more architectural fragments stored thereon, which may each correspond to one or more requirements and / or certain requirement information. The repository 324 may also include indications of the association between architectural fragments and the requirements and / or requirement information associated therewith.

[0121] In these and other embodiments, based on the program architecture 316 and the requirement information 304, the repository query 312 may include determining whether at least one architecture fragment 332 has been identified and / or stored in the repository 324 in association with a requirement corresponding to the requirement information 304. Therefore, such architecture fragments 332 can be identified based on the repository 324 without having to perform architecture hint generation 302 and / or GLM interaction 308, which helps save computational resources.

[0122] Additionally or alternatively, process 300 may include a repository storage operation 314. Repository storage operation 314 may include storing one or more schema fragments 332 in relation to demand information 304 in a repository 324. Thus, repository storage 314 can be added to repository 324 so that future repository queries 312 can use the stored information in future iterations of process 300.

[0123] Without departing from the scope of this disclosure, process 300 may be modified, added to, or deleted. For example, although process 300 is shown as discrete blocks or operations, depending on the specific implementation, the individual blocks or operations of process 300 may be divided into more blocks, combined into fewer blocks, or eliminated. Furthermore, in some embodiments, one or more operations may be combined into fewer operations, or extended to include additional operations.

[0124] Figure 4 An example program interface specification fragment identification process 400 (“Process 400”) according to one or more embodiments of this disclosure is illustrated. Each operation or block of Process 400 described herein may include a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. Process 400 may also be embodied as computer-usable instructions stored on a computer storage medium. Process 400 may be provided by a standalone application, service, or managed service (independently or in combination with other managed services), or as a microservice via an application programming interface (API) or as a plug-in to another product, to name a few. For example, Process 400 may be provided by… Figure 1 The system Figure 9 computing devices and / or Figure 10The process 400 is executed by one or more components in the data center. However, process 400 may be additionally or alternatively executed by any system or combination of systems, including but not limited to the systems described herein. Furthermore, for ease of interpretation, the description of process 400 is concerned with identifying interface specification fragments related to “requirements” associated with software programs; however, such a process can be used to identify interface specification fragments for any number of requirements for any number of software programs. The software program may include any suitable software program, such as any software program described in this disclosure.

[0125] Furthermore, in some embodiments, one or more operations of process 400 may be performed to perform actions related to... Figure 2 The described interface specification fragment identification 212. However, process 400 is not limited to the implementation of interface specification fragment identification 212.

[0126] Process 400 may include an interface hint generation operation 402 (“Interface Hint Generation 402”), which can be used to generate interface hint 406. Interface hint 406 can be generated based on requirement information 404, interface specification 418, and interface hint template 428. Requirement information 404 can be combined with… Figure 2 The requirement information 204 is similar or related, and the interface specification 418 can be similar to it. Figure 2 The interface specification 218 is similar to or similar to the interface prompt template 428. Figure 2 The interface prompt template 228 is similar or analogous.

[0127] In some embodiments, interface prompt generation 402 may include providing requirement information 404 and interface specification 418 as input (e.g., by...). Figure 1 The test module 112 provides the interface prompt template 428. In this disclosure, a reference to "interface specification 418" provided as input can refer to a reference to the interface specification, which can be used to identify where the interface specification 418 is found and / or accessed.

[0128] In some embodiments, interface hint generation 402 may include executing an interface hint template 428 using input requirement information 404 and interface specification 418. Accordingly, the execution of interface hint template 428 may generate interface hint 406 based on the pre-filled language of interface hint template 428 and the input requirement information 404 and interface specification 418.

[0129] In these and other embodiments, process 400 may include GLM interaction operation 408 (“GLM interaction 408”). GLM interaction 408 may include providing interface prompt 406 to GLM 410, which can interact with… Figure 1The GLM 410 is similar to or analogous to the GLM 110 in these and other embodiments. In these and other embodiments, the GLM 410 can identify one or more fragments of the interface specification 418 corresponding to the requirements associated with the requirement information 404 as interface fragments 434. In some embodiments, the GLM 410 can output interface fragments 434. In some embodiments, interface fragments 434 can be associated with... Figure 2 The interface fragment 234 is similar to or similar to that in the previous one.

[0130] Additionally or alternatively, in addition to or as a supplement to Interface Prompt Generation 402 and GLM Interaction 408, process 400 may include a repository query operation 412 (“Repository Query 412”). Repository Query 412 may include accessing repository 424. Repository 424 may be connected to… Figure 1 The repository 124 is similar to or analogous to the one stored therein, and may include one or more lists of one or more interface fragments stored thereon, which may each correspond to one or more requirements and / or certain requirement information. The repository 424 may also include indications of the association between the interface fragments and the requirements and / or requirement information associated therewith.

[0131] In these and other embodiments, based on interface specification 418 and requirement information 404, repository query 412 may include determining whether at least one interface fragment 434 has been identified and / or stored in repository 424 in association with a requirement corresponding to requirement information 404. Therefore, such interface fragment 434 can be identified based on repository 424 without having to perform interface hint generation 402 and / or GLM interaction 408, which helps save computational resources.

[0132] Additionally or alternatively, process 400 may include a repository storage operation 414. Repository storage operation 414 may include storing one or more interface fragments 434 in relation to demand information 404 in a repository 424. Thus, repository storage 414 can be added to repository 424 so that future repository queries 412 can use the stored information in future iterations of process 400.

[0133] Without departing from the scope of this disclosure, process 400 may be modified, added to, or deleted. For example, although process 400 is shown as discrete blocks or operations, depending on the specific implementation, the individual blocks or operations of process 400 may be divided into additional blocks, combined into fewer blocks, or eliminated. Furthermore, in some embodiments, one or more operations may be combined into fewer operations, or extended to include additional operations.

[0134] Furthermore, in some embodiments, the storage library 424 can be connected to... Figure 3The storage unit 324 is the same as that in the previous one. Alternatively, storage unit 424 may be connected to... Figure 3 The storage library 324 in this embodiment differs from that in others. In these and other embodiments, the GLM 410 can be... Figure 3 The GLM 310 is the same as the one in the series. Alternatively, the GLM 410 may be compatible with... Figure 3 It is different from the GLM 310 in the text.

[0135] Figure 5 An example test specification generation process 500 (“Process 500”) according to one or more embodiments of this disclosure is illustrated. Each operation or block of Process 500 described herein may include a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. Process 500 may also be embodied as computer-usable instructions stored on a computer storage medium. Process 500 may be provided by a standalone application, service, or managed service (independently or in combination with other managed services), or as a microservice via an application programming interface (API) or as a plug-in to another product, to name a few. For example, Process 500 may be provided by… Figure 1 The system Figure 9 computing devices and / or Figure 10 The process 500 is performed by one or more components of the data center. However, process 500 may be performed additionally or alternatively by any system or combination of systems, including but not limited to the systems described herein. Furthermore, for ease of interpretation, the description of process 500 pertains to generating test specifications related to “requirements” associated with software programs; however, such a process can be used to generate test specifications for any number of requirements for any number of software programs. Software programs may include any suitable software program, such as any software program described in this disclosure.

[0136] Furthermore, in some embodiments, one or more operations of process 500 may be performed to perform actions related to... Figure 2 The described test specification generation 244. However, process 500 is not limited to the implementation of test specification generation 244.

[0137] Process 500 may include a test specification hint generation operation 502 (“Test Specification Hint Generation 502”), which can be used to generate test specification hints 506. Test specification hints 506 can be generated based on requirement information 504, information fragments 530, and test specification hint templates 540. Requirement information 504 can be combined with… Figure 2 The demand information 204 is similar or related, and information fragment 530 can be related to... Figure 2 Information fragment 230 is similar or related to test specification prompt template 540. Figure 2 The test specification prompt template 240 is similar or analogous.

[0138] In some embodiments, test specification prompt generation 502 may include providing requirement information 504 and information fragment 530 as input (e.g., by...). Figure 1 The test module 112 provides a test specification prompt template 540. In this disclosure, a reference to "information fragment 530" provided as input can refer to a reference to an information fragment that can be used to identify where information fragment 530 is found and / or accessed.

[0139] In some embodiments, test specification hint generation 502 may include executing a test specification hint template 540 using input requirement information 504 and information fragment 530. Therefore, the execution of the test specification hint template 540 may generate a test specification hint 506 based on the pre-filled language of the test specification hint template 540 and the input requirement information 504 and information fragment 530.

[0140] In these and other embodiments, process 500 may include GLM interaction 508 (“GLM interaction 508”). GLM interaction 508 may include providing test specification prompts 506 to GLM 510, which can interact with… Figure 1 The GLM 510 is similar to or analogous to the GLM 110 in these and other embodiments. In these and other embodiments, the GLM 510 can generate a test specification 520 based on test specification hints 506. In some embodiments, the GLM 510 can output the test specification 520. In some embodiments, the test specification 520 can be related to... Figure 2 The test specification 220 is similar to or analogous to that in the test specification.

[0141] Additionally or alternatively, in addition to and / or as a supplement to the test specification prompt generation 502 and GLM interaction 508, process 500 may include a repository query operation 512 (“Repository Query 512”). Repository Query 512 may include accessing repository 524. Repository 524 may be connected to… Figure 1 The repository 124 is similar to or analogous to the one stored therein, and may include one or more lists of test specifications and / or corresponding test cases stored therein, which may each correspond to one or more requirements and / or certain requirement information. The repository 524 may also include indications of the association between the test specifications and / or test cases and the requirements and / or requirement information associated therewith.

[0142] In these and other embodiments, based on requirement information 504, a repository query 512 may include determining whether at least one test case of test specification 520 has been identified and / or stored in repository 524 in association with a requirement corresponding to requirement information 504. Therefore, such test cases can be identified based on repository 524 without having to perform test specification hint generation 502 and / or GLM interaction 508, which helps save computational resources.

[0143] Additionally or alternatively, process 500 may include a repository storage operation 514. Repository storage operation 514 may include storing one or more test cases of test specification 520 in relation to requirement information 504 in repository 524. Therefore, repository storage 514 can be added to repository 524 so that future repository queries 512 can use the stored information in future iterations of process 500.

[0144] Without departing from the scope of this disclosure, process 500 may be modified, added to, or deleted. For example, although the individual blocks or operations of process 500 are shown as discrete blocks or operations, depending on the specific implementation, the individual blocks or operations of process 500 may be divided into additional blocks, combined into fewer blocks, or eliminated. Furthermore, in some embodiments, one or more operations may be combined into fewer operations, or extended to include additional operations.

[0145] Furthermore, in some embodiments, the storage library 524 can be connected to... Figure 3 Storage 324 and / or Figure 4 The storage unit 424 is the same. Additionally or alternatively, the storage unit 524 may be... Figure 3 Storage 324 and / or Figure 4 The storage library 424 is different. In these and other embodiments, the GLM 510 can be with Figure 3 GLM 310 and / or Figure 4 The GLM 410 is identical. Additionally or alternatively, the GLM 510 can be... Figure 3 GLM 310 and / or Figure 4 It is different from the GLM 410.

[0146] Figure 6An example test implementation of process 600 (“Process 600”) according to one or more embodiments of this disclosure is illustrated. Each operation or block of Process 600 described herein may include a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions may be implemented by a processor executing instructions stored in memory. Process 600 may also be embodied as computer-usable instructions stored on a computer storage medium. Process 600 may be provided by a standalone application, service, or managed service (alone or in combination with other managed services), or as a microservice via an application programming interface (API), or as a plug-in to another product, to name a few. Process 600 may be provided by Figure 1 The system shown Figure 9 The computing devices and / or shown Figure 10 The process 600 may be executed by one or more components of the data center shown. However, process 600 may also be executed by any single system or any combination of systems, including but not limited to the systems described herein. Furthermore, for ease of explanation, the description of process 600 pertains to generating test implementations of “requirements” associated with a software program, but this process can also be used to generate test implementations for any number of requirements for any number of software programs. The software program may include any suitable software program, such as any software program described in this disclosure.

[0147] Furthermore, in some embodiments, one or more operations of process 600 may be performed to perform actions related to... Figure 2 The described test implementation generates 246. However, process 600 is not limited to the implementation of test implementation generating 246.

[0148] Process 600 may include a test implementation hint generation operation 602 (“Test Implementation Hint Generation 602”), which can be used to generate a test implementation hint 606. The test implementation hint 606 can be generated based on a test specification 620, one or more information fragments 630, and a test implementation hint template 642. The test specification 620 can be combined with… Figure 2 Test specification 220 is similar to or similar to it, and information fragment 630 may be related to it. Figure 2 The information fragment 230 is similar to or similar to the test implementation method prompt template 642, which can be similar to... Figure 2 The test implementation method is similar to or similar to template 242.

[0149] In some embodiments, test implementation hint generation 602 may include providing test specification 620 and information fragment 630 as input (e.g., by...). Figure 1The test module 112 provides a test implementation hint template 642. In this disclosure, a reference to "information fragment 630" provided as input can refer to a reference to the information fragment, which can be used to identify where the information fragment 630 is found and / or accessed.

[0150] In some embodiments, test implementation hint generation 602 may include executing a test implementation hint template 642 using an input test specification 620 and information fragment 630. Therefore, the execution of the test implementation hint template 642 may generate a test implementation hint 606 based on the pre-filled language of the test implementation hint template 642 and the input test specification 620 and information fragment 630.

[0151] In these and other embodiments, process 600 may include GLM interaction 608 (“GLM interaction 608”). GLM interaction 608 may include providing test implementation hints 606 to GLM 610, which can interact with… Figure 1 The GLM 610 is similar to or analogous to the GLM 110 in these and other embodiments. In these and other embodiments, the GLM 610 can generate a test implementation 622 based on the test implementation hint 606. In some embodiments, the GLM 610 can output the test implementation 622. In some embodiments, the test implementation 622 can be similar to... Figure 2 The test implementation method in 222 is similar or analogous.

[0152] Additionally or alternatively, in addition to and / or as a supplement to the test implementation hint generation 602 and GLM interaction 608, process 600 may include a repository query operation 612 (“Repository Query 612”). Repository Query 612 may include accessing repository 624. Repository 624 may be connected to… Figure 1 The repository 624 is similar to or may include one or more lists of one or more test implementations stored thereon, each of which may correspond to one or more requirements and / or certain requirement information. The repository 624 may also include indications of the association between the test implementations and their associated requirements and / or requirement information. In these and other embodiments, the repository 624 may include indications of the association between the code of the test implementation 622 and different test cases.

[0153] In these and other embodiments, based on test specification 620 and / or information fragment 630, repository query 612 may include determining whether at least one code set corresponding to at least one test case of test specification 620 has been identified and / or stored in repository 624. Therefore, such code sets can be identified based on repository 624 without performing test implementation hint generation 602 and / or GLM interaction 608, which helps save computational resources.

[0154] Additionally or alternatively, process 600 may include a repository storage operation 614. Repository storage operation 614 may include storing one or more code sets of test implementations in association with test cases of their corresponding test specifications 620. Therefore, repository storage 614 can be added to repository 624 so that future repository queries 612 can use the stored information in future iterations of process 500.

[0155] Without departing from the scope of this disclosure, process 600 may be modified, added to, or deleted. For example, although process 600 is shown as discrete blocks or operations, depending on the specific implementation, the individual blocks or operations of process 600 may be divided into additional blocks, combined into fewer blocks, or eliminated. Furthermore, in some embodiments, one or more operations may be combined into fewer operations, or extended to include additional operations.

[0156] Furthermore, in some embodiments, the storage library 624 can be connected to... Figure 3 Storage 324 Figure 4 Storage 424 and / or Figure 5 The storage unit 524 is the same. Additionally or alternatively, the storage unit 524 may be... Figure 3 Storage 324 Figure 4 Storage 424 and / or Figure 5 The storage library 524 is different. In these and other embodiments, the GLM 610 can be with Figure 3 GLM310 in Figure 4 GLM 410 and / or Figure 5 The GLM 510 is the same as the GLM 510. Alternatively, the GLM 610 may be compatible with... Figure 3 GLM 310 in Figure 4 GLM 410 and / or Figure 5 It is different from the GLM 510 in the text.

[0157] Now for reference Figure 7Each block of the method 700 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. The method can also be embodied as computer-usable instructions stored on a computer storage medium. The method can be provided by a standalone application, service, or managed service (independently or in combination with other managed services), or as a microservice via an application programming interface (API) or as a plug-in to another product, to name a few. For example, method 700 can be provided by… Figure 1 The system shown Figure 9 The computing devices and / or shown Figure 10 The method can be performed by one or more components of the data center shown. However, the method can also be performed by any single system or any combination of systems, including but not limited to the systems described herein.

[0158] Figure 7 This is a flowchart of a method 700 for generating a test architecture for a software program according to one or more embodiments of the present disclosure. Method 700 includes, at block B702, generating one or more GLM hints associated with the test software program. The one or more GLM hints may be based on at least one or more hint templates, which are populated at least based on program information describing the software program and requirement information describing the requirements of the software program. In some embodiments, the hints may be based on the information regarding the present disclosure... Figures 1 to 6 Describe one or more operations to generate one or more prompts.

[0159] At block B704, a test architecture corresponding to the software program's fulfillment requirements can be generated based on one or more GLM outputs corresponding to GLM hints. In some embodiments, it can be based on the information provided in this disclosure regarding... Figures 1 to 6 The described one or more operations generate a test architecture.

[0160] At block B706, in some embodiments, the software program can be tested based on a test architecture. In some embodiments, the test can be based on the information provided in this disclosure regarding... Figure 1 and Figure 2 Describe one or more operations.

[0161] Method 700 may be modified, added to, or deleted without departing from the scope of this disclosure. For example, although the individual blocks of method 700 are shown as discrete blocks, they may be divided into additional blocks, combined into fewer blocks, or deleted, depending on the specific implementation. Furthermore, in some embodiments, method 700 can be used to perform multiple different authentications on multiple different peripheral devices.

[0162] Example language model

[0163] In at least some embodiments, language models such as Large Language Models (LLM), Visual Language Models (VLM), Multimodal Language Models (MMLM), and / or other types of generative artificial intelligence (AI) can be implemented. These models may be able to understand, summarize, translate, and / or otherwise generate text (e.g., natural language text, code, etc.), images, videos, computer-aided design (CAD) assets, OMNIVERSE and / or METAVERSE file information (e.g., USD formats such as OpenUSD), and / or the like based on context provided in input prompts or queries. In embodiments, these language models may be considered “large” because they are trained on massive datasets and have architectures with a large number of learnable network parameters (weights and biases)—e.g., millions or billions of parameters. LLM / VLM / MMLM / etc. can be implemented for summarizing textual data, analyzing data (e.g., text, images, videos, etc.), extracting insights from data (e.g., text, images, videos, etc.), and generating new text / images / videos / etc. in a user-specified style, tone, and / or format. In some embodiments, the LLM / VLM / MMLM / etc. disclosed herein may be specifically designed for text processing, while in others, a multimodal LLM may be implemented to accept, understand, and / or generate text and / or other types of content, such as images, audio, 2D and / or 3D data (e.g., USD format) and / or video. For example, a Visual Language Model (VLM) or more specifically a Multimodal Language Model (MMLM) may be implemented to accept images, video, audio, text, 3D designs (e.g., CAD) and / or other input data types and / or generate or output images, video, audio, text, 3D designs and / or other output data types.

[0164] Various types of LLM / VLM / MMLM / etc. architectures can be implemented in various embodiments. For example, different architectures can be implemented using different techniques to understand and generate outputs (e.g., text, audio, video, images, 2D and / or 3D design or asset data, etc.). In some embodiments, LLM / VLM / MMLM / etc. architectures (e.g., recurrent neural networks (RNNs) or long short-term memory networks (LSTMs)) can be used, while in other embodiments, converter architectures (e.g., architectures relying on self-attention and / or cross-attention (e.g., between contextual data and textual data) mechanisms) can be used to understand and recognize relationships between words or tokens and / or contextual data (e.g., other text, video, images, design data, USD, etc.). One or more generative processing pipelines including LLM / VLM / MMLM / etc. may also include one or more diffusion blocks (e.g., noise reduction blocks). The LLM / VLM / MMLM / etc. of this disclosure may include encoder and / or decoder blocks. For example, discriminative or encoder-only models (e.g., BERT (Bidirectional Encoder Representations from Transformers)) can be implemented for tasks involving language understanding (e.g., classification, sentiment analysis, question answering, and named entity recognition). As another example, generative or decoder-only models (e.g., GPT (Generative Pretrained Transformer)) can be implemented for tasks involving language and content generation (e.g., text completion, story generation, and dialogue generation). LLM / VLM / MMLM / etc., including encoder and decoder components (e.g., T5 (Text-to-Text Transformer)), can be implemented to understand and generate content, such as for translation and summarization. These examples are not intended to be limiting and any architecture type (including, but not limited to, those described herein) can be implemented depending on the specific implementation and the task performed using LLM / VLM / MMLM / etc.

[0165] In various embodiments, LLM / VLM / MMLM / etc. can be trained using unsupervised learning, whereby LLM / VLM / MMLM / etc. learns patterns from a large amount of unlabeled text / audio / video / image / design / USD / etc. data. Due to the extensive training, in these embodiments, the model may not require task-specific or domain-specific training. An LLM / VLM / MMLM / etc. extensively pre-trained on a large amount of unlabeled data can be referred to as a base model and can excel at various tasks, such as question answering, summarizing, filling in missing information, translation, and image / video / design / USD / data generation. Some LLM / VLM / MMLM / etc. can be customized for specific use cases using techniques such as cue tuning, fine-tuning, retrieval augmentation generation (RAG), adding adapters (e.g., custom neural networks and / or neural network layers to tune or adjust cues or labels to bias the language model towards a specific task or domain), and / or using optimization models for specific tasks and / or other fine-tuning or customization techniques within a specific domain.

[0166] In some embodiments, the LLM / VLM / MMLM / etc. disclosed herein can be implemented using various model alignment techniques. For example, in some embodiments, guardrails can be implemented to identify incorrect or unwanted inputs (e.g., prompts) and / or outputs of the model. In this process, the system can use guardrails and / or other model alignment techniques to prevent the processing of specific unwanted inputs using LLM / VLM / MMLM / etc., and / or to prevent the output or presentation of information generated by LLM / VLM / MMLM / etc. (e.g., displays, audio outputs, etc.). In some embodiments, one or more additional models (or layers thereof) can be implemented to identify problems with the model's inputs and / or outputs. For example, these "protective" models can be trained to identify "safe" or otherwise okay or desired inputs and / or outputs and / or "unsafe" or otherwise unwanted inputs and / or outputs for a particular application / implementation. Therefore, the LLM / VLM / MMLM / etc. disclosed herein are unlikely to output language / text / audio / video / design data / USD data / etc. that may be offensive, vulgar, inappropriate, insecure, out of scope, and / or unwanted for a particular application / implementation.

[0167] In some embodiments, an LLM / VLM / etc. can be configured or able to access or use one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc. For example, for certain tasks or operations where the model is not ideally suited, the model may have instructions for accessing one or more plugins (e.g., third-party plugins) to help process the current input (e.g., as a result of training, and / or based on instructions in a given prompt). In such an example, when at least part of the prompt relates to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs) to retrieve relevant information. Another example is that if at least part of the response requires mathematical computation, the model can access one or more mathematical plugins or APIs to help solve the problem, and then the response from the plugins and / or APIs can be used in the model's output. This process can be repeated (e.g., recursively) an arbitrary number of iterations, using any number of plugins and / or APIs, until a response to each query / question / request / process / action / etc. can be generated in response to the input prompt. Therefore, models can rely not only on their own knowledge gained from training on large datasets, but also on the expertise or optimized properties of one or more external resources (such as APIs, plugins, etc.).

[0168] In some embodiments, multiple language models (e.g., LLM / VLM / MMLM / etc., multiple instances of the same language model, and / or multiple hints provided to the same language model or instances of the same language model) can be implemented, executed, or accessed (e.g., using one or more plugins, user interfaces, APIs, databases, data stores, repositories, etc.) to provide output in response to the same query or in response to separate parts of a query. In at least one embodiment, the same input query and hints (e.g., a set of constraints, condition generators, etc.) can be provided to multiple language models (e.g., language models with different architectures, language models trained on different (e.g., updated) data corpora). In one or more embodiments, the language models can be different versions of the same base model. In one or more embodiments, at least one language model can be instantiated as multiple agents, for example, providing more than one hint to constrain, guide, or otherwise influence the style, content, or character of the provided output. In one or more exemplary non-limiting embodiments, the same language model can be required to provide output corresponding to different roles, perspectives, characters, or different knowledge bases, as defined by the provided hints.

[0169] In any such embodiment, the outputs of two or more (e.g., each) language models, two or more versions of at least one language model, two or more instantiated proxies of at least one language model, and / or provided to two or more prompts for at least one language model can be further processed, such as aggregated, compared, or filtered, or used to determine (and provide) a consensus response. In one or more embodiments, the output from one language model (or version, instance, or proxy) can be provided as input to another language model for further processing and / or validation. In one or more embodiments, the language model can be required to generate or otherwise obtain output about the input source material, wherein the output is associated with the input source material. This association may include, for example, generating captions or text portions embedded (e.g., as metadata) within the input source text or image. In one or more embodiments, the output of the language model can be used to determine the validity of the input source material for further processing or inclusion in a dataset. For example, the language model can be used to evaluate the presence (or absence) of a target word in a text portion or the presence (or absence) of an object in an image, wherein the text or image is annotated to indicate such presence (or absence). Alternatively, the determination from the language model can be used to determine whether the source material should be included in the curatorial dataset, for example, but not limited to this.

[0170] Figure 8A This is a block diagram of an example generative language model system 800 suitable for implementing at least some embodiments of the present disclosure. Figure 8A In the example shown, the generative language model system 800 includes a retrieval augmentation generation (RAG) component 892, an input processor 805, a tokenizer 810, an embedding component 820, a plug-in / API 895, and a generative language model (LM) 830 (which may include LLM, VLM, multimodal LM, etc.).

[0171] At a high level, the input processor 805 can receive input 801, which includes text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasound, etc.), 3D design data, CAD data, generic scene descriptor (USD) data (e.g., OpenUSD, etc.), depending on the architecture of the generative LM 830 (e.g., LLM / VLM / MMLM, etc.). In some embodiments, input 801 includes plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, input 801 may include numerical sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., tabular format, JSON, or XML). In generative LM In some implementations of 830 capable of handling multimodal input, input 801 can combine text (or text that may be omitted) with image data, audio data, video data, design data, USD data, and / or other types of input data (e.g., but not limited to the data described herein). Taking raw input text as an example, input processor 805 can prepare the raw input text in various ways. For example, input processor 805 can perform various types of text filtering to remove noise from relevant text content (e.g., special characters, punctuation marks, HTML tags, stop words, portions of images, portions of audio, etc.). In examples involving stop words (common words that often have little semantic meaning), input processor 805 can remove stop words to reduce noise and allow the generative LM 830 to focus on more meaningful content. Input processor 805 can apply text normalization, for example, by converting all characters to lowercase, removing accent marks, and / or handling special cases (such as abbreviations or abbreviations) to ensure consistency. These are just a few examples; other types of input processing can be applied.

[0172] In some embodiments, RAG component 892 (which may include one or more RAG models, and / or may be performed using generative LM 830 itself) may be used to retrieve additional information to be used as part of input 801 or a prompt. RAGs can be used to enhance input to LLM / VLM / MMLM / etc. with external knowledge to make the answer to a specific question or query or request more relevant, for example, where specific knowledge is required. RAG component 892 may obtain this additional information from one or more external sources (e.g., basic information such as basic text / images / videos / audio / USD / CAD / etc.), which can then be fed along with the prompt to LLM / VLM / MMLM / etc. to improve the accuracy of the model's response or output.

[0173] For example, in some embodiments, in addition to the data retrieved using RAG component 892, input 801 may also be generated using query or model input (e.g., questions, requests, etc.). In some embodiments, input processor 805 may analyze input 801 and communicate with RAG component 892 (or in embodiments, RAG component 892 may be part of input processor 805) to identify relevant text and / or other data to provide to generative LM 830 as additional context or information source, typically from which to identify responses, answers, or outputs 890. For example, when the input indicates that a user is interested in the required tire pressure for a particular brand and model of vehicle, RAG component 892 may use a RAG model, for example, to perform a vector search in the embedding space to retrieve tire pressure information or its corresponding text from a digital (embedded) version of the owner's manual for that particular vehicle brand and model. Similarly, when a user revisits the chatbot related to a specific product sale or service, the RAG component 892 can retrieve previously stored conversation history (or at least its summary) and provide the previous conversation history, along with the current inquiry / request, as part of the generative LM 830 as input 801.

[0174] RAG component 892 can use various RAG techniques. For example, it can use naive RAG ( The document is indexed, chunked, and applied to an embedding model to generate embeddings corresponding to chunks. User queries can also be applied to this embedding model and / or another embedding model of the RAG component 892, and the embeddings of the chunks can be compared with the embeddings of the query to identify the most similar / relevant embeddings to the query. These most similar / relevant embeddings can be provided to the generative LM 830 to generate output.

[0175] In some embodiments, more advanced RAG techniques can be used. For example, chunks can undergo pre-retrieval processes (e.g., routing, rewriting, metadata analysis, expansion, etc.) before being passed to the embedding model. Furthermore, post-retrieval processes (e.g., re-ranking, hint compression, etc.) can be performed on the output of the embedding model before generating the final embedding, which is then used for comparison with the input query.

[0176] As a further example, modular RAG techniques can be used, such as those similar to Naive RAG and / or Advanced RAG, but also including features such as hybrid search, recursive retrieval and query engines, StepBack methods, subqueries and hypothetical document embeddings.

[0177] As another example, Graph RAG can use a knowledge graph as a source of context or factual information. Graph RAG can be implemented using a graph database as a source of contextual information sent to LLM / VLM / MMLM / etc. Instead of providing the model with data chunks extracted from larger documents (which may result in a lack of context, factual accuracy, linguistic accuracy, etc.) (or anything other than providing the model with data chunks extracted from larger documents), Graph RAG can also provide structured entity information to LLM / VLM / MMLM / etc. by combining structured entity text descriptions with their many attributes and relationships, thus giving the model deeper insights. In implementing Graph RAG, the systems and methods described herein use graphs as content stores and extract relevant document chunks, requiring LLM / VLM / MMLM / etc. to use them to answer questions. In such embodiments, the knowledge graph may contain relevant textual content and metadata about the knowledge graph, or it may be integrated with a vector database. In some embodiments, Graph RAG can use the graph as a subject matter expert, where descriptions of concepts and entities relevant to the query / hint can be extracted and passed to the model as semantic context. These descriptions may include relationships between concepts. In other examples, the graph can be used as a database where a portion of a query / hint can be mapped to a graph query, the graph query can be executed, and LLM / VLM / MMLM / etc. can aggregate the results. In such examples, the graph can store relevant factual information and can be used for queries (natural language queries) and entity links to graph query tools (NL to graph query tools). In some embodiments, the graph RAG (e.g., using a graph database) can be combined with standard (e.g., vector database) RAGs and / or other RAG types to benefit from a variety of approaches.

[0178] In any embodiment, the RAG component 892 can implement plugins, APIs, user interfaces, and / or other functions to perform RAG. For example, LLM / VLM / MMLM / etc. can use graph RAG plugins to run queries on knowledge graphs to extract relevant information to feed into the model, and can use standard or vector RAG plugins to run queries on vector databases. For example, the graph database can interact with the plugin's REST interface, thus decoupling the graph database from the vector database and / or the embedded model.

[0179] The tokenizer 810 can segment (e.g., processed) text data into smaller units (tags) for subsequent analysis and processing. Depending on the implementation, the tags can represent individual words, sub-words, characters, audio / video / images, etc. Word-based tokenization divides the text into individual words, treating each word as a separate tag. Sub-word tokenization breaks words down into smaller meaningful units (e.g., prefixes, suffixes, stems), enabling the generative LM 830 to understand morphological changes and process words outside the vocabulary more effectively. Character-based tokenization represents each character as a separate tag, enabling the generative LM 830 to process text at a fine-grained level. The choice of tokenization strategy can depend on factors such as the language being processed, the task at hand, and / or the characteristics of the training dataset. Therefore, the tokenizer 810 can transform (e.g., processed) text into a structured format according to the tokenization scheme implemented in a particular embodiment.

[0180] Embedding component 820 can use any known embedding technique to transform discrete tokens into semantically meaningful (e.g., dense, continuous vector) representations. For example, embedding component 820 can use pre-trained word embeddings (e.g., Word2Vec, GloVe, or FastText), one-hot encoding, Term Frequency-Inverse Document Frequency (TF-IDF) encoding, one or more embedding layers of a neural network, and / or others.

[0181] In some implementations where input 801 includes image data / video data, etc., input processor 801 may resize the data to a standard size compatible with the format of the corresponding input channel and / or normalize pixel values ​​to a common range (e.g., 0 to 1) to ensure consistent representation, and embedding component 820 may encode the image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations where input 801 includes audio data, input processor 801 may resample the audio file to a consistent sampling rate for uniform processing, and embedding component 820 may use any known technique to extract and encode audio features, such as in the form of a spectrogram (e.g., a Mel spectrogram). In some implementations where input 801 includes video data, input processor 801 may extract frames or apply resizing to extracted frames, and embedding component 820 may extract features such as optical flow embedding or video embedding and / or encode temporal information or frame sequences. In some implementations where input 801 includes multimodal data, the embedded component 820 can use techniques such as early fusion (concatenation), late fusion (sequential processing), and attention-based fusion (e.g., self-attention, cross-attention) to fuse representations of different types of data (e.g., text, images, audio, data, video, design, etc.).

[0182] Other components of the generative LM 830 and / or generative LM system 800 may use different types of neural network architectures depending on the implementation scheme. For example, a transducer-based architecture (such as the one used in models like GPT) may be implemented, and it may include a self-attention mechanism that weights the importance of different words or tokens in the input sequence and / or a feedforward network that processes the output of the self-attention layer, applying a nonlinear transformation to the input representation and extracting higher-level features. Some non-limiting example architectures include transducers (e.g., encoder-decoder, decoder-only, multimodal), RNNs, LSTMs, fusion models, diffusion models, cross-modal embedding models that learn a joint embedding space, graph neural networks (GNNs), hybrid architectures that combine different types of adversarial networks (such as generative adversarial networks or GANs or adversarial autoencoders (AAEs) for joint distribution learning), etc. Therefore, depending on the implementation scheme and architecture, the embedded component 820 can apply the encoded representation of the input 801 to the generative LM 830, and the generative LM 830 can process the encoded representation of the input 801 to generate an output 890, which may include response text and / or other types of data.

[0183] As described herein, in some embodiments, the generative LM 830 may be configured to access or use (or be able to access or use) plugins / APIs 895 (which may include one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc.). For example, for certain tasks or operations where the generative LM 830 is not ideally suited, the model may have instructions (e.g., as a result of training, and / or based on instructions in a given prompt, such as instructions retrieved using RAG component 892) to access one or more plugins / APIs 895 (e.g., third-party plugins) to help process the current input. In such an example, when at least part of the prompt is related to a restaurant or weather, the model may access one or more restaurant or weather plugins (e.g., via one or more APIs), sending at least part of the prompt related to a particular plugin / API 895 to the plugin / API 895, which may process the information and return an answer to the generative LM 830, which may use the response to generate output 890. This process can be repeated (e.g., recursively) an arbitrary number of iterations and repeated with any number of plugins / APIs 895 until an output 890 that resolves each query / question / request / process / action / etc. from input 801 is generated. Therefore, the model can rely not only on its own knowledge gained from training on a large dataset and / or from data retrieved using the RAG component 892, but also on the expertise or optimized properties of one or more external resources (e.g., plugins / APIs 895).

[0184] Figure 8B This is a block diagram of an example implementation scheme, where the generative LM 830 includes a converter encoder-decoder. For example, suppose the input text (e.g., “Who discovered gravity”) is tokenized (e.g., by...) Figure 8A The tokenizer 810) is used for tokens such as words, and each token is encoded (e.g., by...). Figure 8A The embedding component 820 is a corresponding embedding (e.g., of size 512). Since these token embeddings do not typically represent the position of the tokens in the input sequence, positional encoding can be added to each token embedding using any known technique to encode the order relation and context of the tokens in the input sequence. Thus, (e.g., the resulting) embeddings can be applied to one or more encoders 835 of the generative LM 830.

[0185] In the example implementation, encoder 835 forms an encoder stack, where each encoder includes a self-attention layer and a feedforward network. In the example converter architecture, each token (e.g., a word) flows through a separate path. Therefore, each encoder can accept a sequence of vectors, pass each vector through the self-attention layer, then through the feedforward network, and then up to the next encoder in the stack. Any known self-attention technique can be used. For example, to compute a self-attention score for each token (word), a query vector, a key vector, and a value vector can be created for each token. The self-attention score for a token pair can be computed by taking the dot product of the query vector and the corresponding key vector, normalizing the resulting score, multiplying by the corresponding value vector, and summing the weighted value vectors. The encoder can apply multi-head attention, where the attention mechanism is applied multiple times in parallel with different learned weight matrices. Any number of encoders can be cascaded to generate a context vector encoding the input. The attention projection layer 840 can transform the context vector into an attention vector (key and value) for the decoder 845.

[0186] In the example implementation, decoder 845 forms a decoder stack, where each decoder includes a self-attention layer, an encoder-decoder self-attention layer that uses attention vectors (keys and values) from the encoder to focus on relevant parts of the input sequence, and a feedforward network. Similar to encoder 835, in the example converter architecture, each token (e.g., a word) flows through a separate path in decoder 845. During the first pass, decoder 845, classifier 850, and generation mechanism 855 can generate a first token, and generation mechanism 855 can apply the generated token as input during a second pass. This process can be repeated cyclically, generating tokens (e.g., words) and adding them to the output of the previous pass, and in subsequent passes applying token embeddings of positionally encoded composite sequences as input to decoder 845, generating one token at a time (called autoregression) until a symbol or token representing the end of the response is predicted. In each decoder, the self-attention layer is typically restricted to focusing only on preceding positions in the output sequence by applying a masking technique (e.g., setting future positions to negative infinity) before the softmax operation. In the example implementation, the encoder-decoder attention layer operates similarly to the (e.g., multi-head) self-attention operation in the encoder 835, except that it creates its queries from the layers below it and obtains keys and values ​​(e.g., matrices) from the output of the encoder 835.

[0187] Therefore, decoder 845 can output some decoded (e.g., vector) representation of the input applied during a particular pass. Classifier 850 can include a multi-class classifier comprising one or more neural network layers and a softmax operation that transforms logit probabilities into probabilities, the neural network layers projecting the decoded (e.g., vector) representation onto corresponding dimensions (e.g., one dimension for each supported word or token in the output vocabulary). Thus, generation mechanism 855 can select or sample words or tokens based on corresponding predicted probabilities (e.g., selecting the word with the highest predicted probability) and append it to the output of the previous pass, thereby generating each word or token sequentially. Generation mechanism 855 can repeat this process, triggering successive decoder inputs and corresponding predictions until a symbol or token representing the end of the response is selected or sampled, at which point generation mechanism 855 can output the generated response.

[0188] Figure 8C This is a block diagram of an example implementation where the generative LM 830 includes a decoder-only converter architecture. For example, Figure 8C The decoder 860 can be used with Figure 8B The decoder 845 operates similarly, except... Figure 8C Each decoder 860 omits the encoder-decoder self-attention layer (because there is no encoder in this implementation). Therefore, decoders 860 can form a decoder stack, where each decoder includes a self-attention layer and a feedforward network. Furthermore, instead of encoding the input sequence, a symbol or tag indicating the end of the input sequence (or the beginning of the output sequence) can be appended to the input sequence, and the resulting sequence (e.g., a corresponding embedding with positional encoding) can be applied to decoder 860. Figure 8B Similar to decoder 845, each tag (e.g., a word) can flow through a separate path in decoder 860, and decoder 860, classifier 865, and generation mechanism 870 can use autoregression to generate one tag at a time sequentially until a symbol or tag indicating the end of the response is predicted. Classifier 865 and generation mechanism 870 can be combined with... Figure 8B The classifier 850 and the generation mechanism 855 operate similarly, wherein the generation mechanism 870 selects or samples each consecutive output label based on the corresponding predicted probability and appends it to the output of the previous iteration, generating each label sequentially until a symbol or label representing the end of the response is selected or sampled. The architectures described herein, and others, are merely examples, and other suitable architectures may be implemented within the scope of this disclosure.

[0189] Example computing device

[0190] Figure 9This is a block diagram of an example computing device 900 suitable for implementing some embodiments of the present disclosure. The computing device 900 may include an interconnect system 902 directly or indirectly coupled to: a memory 904, one or more central processing units (CPUs) 906, one or more graphics processing units (GPUs) 908, a communication interface 910, input / output (I / O) ports 912, input / output components 914, a power supply 916, one or more presentation components 918 (e.g., one or more displays), and one or more logic units 920. In at least one embodiment, one or more computing devices 900 may include one or more virtual machines (VMs), and / or any of their components may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 908 may include one or more vGPUs, one or more CPUs 906 may include one or more vCPUs, and / or one or more logic units 920 may include one or more virtual logic units. Thus, one or more computing devices 900 may include discrete components (e.g., a full GPU dedicated to computing device 900), virtual components (e.g., a portion of the GPU dedicated to computing device 900), or a combination thereof.

[0191] although Figure 9 The various blocks are shown as connected via interconnect system 902 using lines, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 918 (such as a display device) may be considered I / O component 914 (e.g., if the display is a touchscreen). As another example, CPU 906 and / or GPU 908 may include memory (e.g., memory 904 may represent a storage device other than the memory of GPU 908, CPU 906, and / or other components). Therefore, Figure 9 The computing devices described are for illustrative purposes only. No distinction is made between such categories as “workstation,” “server,” “laptop computer,” “desktop computer,” “tablet computer,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of… Figure 9 Within the scope of computing devices.

[0192] Interconnect system 902 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 902 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 906 may be directly connected to memory 904. Further, CPU 906 may be directly connected to GPU 908. In cases where there is a direct or point-to-point connection between components, interconnect system 902 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required to be included in computing device 900.

[0193] The memory 904 may include any computer-readable medium from a variety of computer-readable media. A computer-readable medium may be any available medium accessible by the computing device 900. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0194] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 904 may store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, Digital Universal Disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 900. As used herein, computer storage media does not include the signal itself.

[0195] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and includes any information transmission medium. The term "modulated data signal" can refer to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the above should also be included within the scope of computer-readable media.

[0196] CPU 906 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. Each CPU 906 may contain one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling numerous software threads simultaneously. CPU 906 may contain any type of processor and may contain different types of processors depending on the type of computing device 900 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 900, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (such as math coprocessors), computing device 900 may also include one or more CPUs 906.

[0197] In addition to or in lieu of one or more CPUs 906, one or more GPUs 908 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. One or more GPUs 908 may be integrated GPUs (e.g., having one or more CPUs 906) and / or one or more GPUs 908 may be discrete GPUs. In embodiments, one or more GPUs 908 may be coprocessors of one or more CPUs 906. GPUs 908 may be used by computing device 900 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 908 may be used for general-purpose computing on a GPU (GPGPU). GPUs 908 may include hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. GPUs 908 may generate pixel data for an output image in response to rendering commands (e.g., rendering commands received from CPUs 906 via a host interface). GPU 908 may include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 904. GPU 908 may include two or more GPUs operating in parallel (e.g., via links). The links may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 908 may produce pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for an analog image). Each GPU may include its own memory or may share memory with other GPUs.

[0198] In addition to or in lieu of CPU 906 and / or GPU 908, logic unit 920 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 906, one or more GPUs 908, and / or one or more logic units 920 may execute any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 920 may be a portion of one or more CPUs 906 and / or GPUs 908 and / or integrated into one or more CPUs 906 and / or GPUs 908, and / or one or more logic units 920 may be discrete components or otherwise external to CPUs 906 and / or GPUs 908. In embodiments, one or more logic units 920 may be coprocessors of one or more CPUs 906 and / or GPUs 908.

[0199] Examples of logic unit 920 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree lateral unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), programmable vision accelerator (PVA) (which may include one or more direct memory access (DMA) systems), one or more vision or vector processing units (VPU), and one or more pixel processing engines (PPE) (e.g.) Examples include 2D arrays of processing elements (each of which communicates north, south, east, and west with one or more other processing elements in the array), one or more decoupled accelerators or units (e.g., decoupled lookup table (DLUT) accelerators or units), vision processing units (VPUs), optical flow accelerators (OFAs), field-programmable gate arrays (FPGAs), neuromorphic chips, quantum processing units (QPUs), associative processing units (APUs), arithmetic logic units (ALUs), application-specific integrated circuits (ASICs), floating-point units (FPUs), input / output (I / O) elements, peripheral component interconnects (PCIs) or fast peripheral component interconnects (PCIe) elements, etc.

[0200] The communication interface 910 may include one or more receivers, transmitters, and / or transceivers enabling the computing device 900 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). The communication interface 910 may include components and functions for enabling communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wirelessband), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 920 and / or the communication interface 910 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via interconnect system 902 to one or more GPUs 908 (e.g., memory of one or more GPUs 908).

[0201] I / O port 912 enables computing device 900 to be logically coupled to other devices including I / O component 914, one or more presentation components 918, and / or other components, some of which may be built into (e.g., integrated into) computing device 900. Illustrative I / O component 914 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dish antennas, scanners, printers, wireless devices, etc. I / O component 914 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by the user. In some cases, input may be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, pen recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of computing device 900. Computing device 900 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. Additionally, the computing device 900 may include an accelerometer or gyroscope that enables motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the computing device 900 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0202] Power supply 916 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 916 may provide power to computing device 900 so that the components of computing device 900 can operate.

[0203] The presentation component 918 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 918 may receive data from other components (e.g., GPU 908, CPU 906, etc.) and output the data (e.g., as images, videos, sounds, etc.).

[0204] Example Data Center

[0205] Figure 10 An example data center 1000 that may be used in at least one embodiment of this disclosure is shown. The data center 1000 may include a data center infrastructure layer 1010, a framework layer 1020, a software layer 1030, and / or an application layer 1040.

[0206] like Figure 10 As shown, the data center infrastructure layer 1010 may include a resource coordinator 1012, grouped computing resources 1014, and node computing resources (“nodes CRs”) 1016(1)-1016(N), where “N” represents any complete positive integer. In at least one embodiment, nodes CRs 1016(1)-1016(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules and / or cooling modules, etc. In some embodiments, one or more node CRs from nodes CRs 1016(1)-1016(N) may correspond to servers having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CRs 1016(1)-1016(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more nodes CRs 1016(1)-1016(N) may correspond to virtual machines (VMs).

[0207] In at least one embodiment, the grouped computing resources 1014 may include individual groups of node CRs 1016 housed within one or more racks (not shown), or a plurality of racks housed within a data center in different geographical locations (also not shown). Individual groups of node CRs 1016 within the grouped computing resources 1014 may include grouped computing, networking, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, a plurality of node CRs 1016, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0208] Resource coordinator 1022 may be configured or otherwise control one or more nodes CRs 1016(1)-1016(N) and / or grouped computing resources 1014. In at least one embodiment, resource coordinator 1022 may include a Software Design Infrastructure (“SDI”) management entity for data center 1000. Resource coordinator 1022 may include hardware, software, or some combination thereof.

[0209] In at least one embodiment, such as Figure 10 As shown, framework layer 1020 may include job scheduler 1028, configuration manager 1034, resource manager 1036, and / or distributed file system 1138. Framework layer 1020 may include a framework for software 1032 supporting software layer 1030 and / or one or more applications 1042 supporting application layer 1040. Software 1032 or application 1042 may respectively contain web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 1020 may be, but is not limited to, free and open-source software web application frameworks (such as Apache Spark) that can utilize distributed file system 1138 for large-scale data processing (e.g., "big data"). TM(Hereinafter referred to as "Spark") is a type of resource. In at least one embodiment, the job scheduler 1028 may include Spark drivers to facilitate the scheduling of workloads supported by different layers of data center 1000. The configuration manager 1034 may be able to configure different layers, such as the software layer 1030 and the framework layer 1020 (which includes Spark and a distributed file system 1138 for supporting large-scale data processing). The resource manager 1036 may be able to manage computing resources mapped to or allocated to clusters of distributed file system 1138 and job scheduler 1028 or to support clusters of distributed file system 1138 and job scheduler 1028. In at least one embodiment, the clustered or grouped computing resources may include grouped computing resources 1014 in the data center infrastructure layer 1010. The resource manager 1036 may coordinate with the resource coordinator 1012 to manage these mapped or allocated computing resources.

[0210] In at least one embodiment, the software 1032 included in the software layer 1030 may include software used in at least a portion of the nodes CRs 1016(1)-1016(N), the grouped computing resources 1014, and / or the distributed file system 1138 of the framework layer 1020. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0211] In at least one embodiment, the application 1042 included in the application layer 1040 may include one or more types of applications used at least in part by nodes CRs 1016(1)-1016(N), grouped computing resources 1014, and / or the distributed file system 1138 of the framework layer 1020. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in combination with one or more embodiments.

[0212] In at least one embodiment, any of the configuration manager 1034, resource manager 1036, and resource coordinator 1012 can implement any number and type of self-modification actions based on any amount and type of data obtained in any technically feasible manner. Self-modification actions can free the data center operator of data center 1000 from making potentially poor configuration decisions and may prevent underutilization and / or poor performance of the data center.

[0213] According to one or more embodiments described herein, data center 1000 may include tools, services, software, or other resources to train one or more machine learning models or to use one or more machine learning models to predict or infer information. For example, one or more machine learning models can be trained by using the software and / or computing resources described above with respect to data center 1000 to compute weight parameters according to a neural network architecture. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks can be used to infer or predict information using the resources described above with respect to data center 1000 by using weight parameters computed through one or more training techniques (such as, but not limited to, those described herein).

[0214] In at least one embodiment, the data center 1000 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the software and / or hardware resources described above may be configured to allow a user to train or perform services that infer information, such as image recognition, speech recognition, or other artificial intelligence services.

[0215] Example network environment

[0216] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 9 It is implemented on one or more instances of one or more computing devices 900—for example, each device may include similar components, features, and / or functions of one or more computing devices 900. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of a data center 1000, examples of which are described in this document. Figure 10 To describe in more detail.

[0217] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0218] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein for the server can be implemented on any number of client devices.

[0219] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or application may respectively include network-based service software or applications. In embodiments, one or more client devices may use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open-source software network application framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0220] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers at the state, region, country, global, etc.). The core server may assign at least a portion of the functionality to the edge server if the connection to the user (e.g., a client device) is relatively close to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0221] One or more client devices may include the information described in this article. Figure 9At least some of the components, features, and functions of one or more example computing devices 900 described. By way of example and not limitation, the client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0222] This disclosure can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions, such as program modules, that are executed by a computer or other machine (such as a personal data assistant or other handheld device). Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure can also be practiced in distributed computing environments, where tasks are performed by remote processing devices linked via a communication network.

[0223] Example autonomous vehicles

[0224] Figure 11AThis is an illustration of an example autonomous vehicle 1100 according to some embodiments of the present disclosure. The autonomous vehicle 1100 (which may alternatively be referred to herein as “vehicle 1100”) may include, but is not limited to, passenger vehicles such as cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police vehicles, ambulances, boats, construction vehicles, underwater vehicles, drones, and / or other types of vehicles (e.g., driverless and / or vehicles accommodating one or more passengers). Autonomous vehicles are generally described according to the level of automation defined by a branch of the U.S. Department of Transportation—the National Highway Traffic Safety Administration (NHTSA)—and the Society of Automotive Engineers (SAE) in its “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of that standard). Vehicle 1100 may be able to perform one or more functions corresponding to Level 3-5 of autonomous driving. Vehicle 1100 may be able to perform one or more functions at Level 1-5 of autonomous driving. For example, depending on the embodiment, vehicle 1100 may be able to perform driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). The term “autonomy” as used herein may include any and / or all types of autonomy of vehicle 1100 or other machines, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, assisted autonomy, semi-autonomy, primary autonomy, or other specified autonomy.

[0225] Vehicle 1100 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 1100 may include a propulsion system 1150, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. Propulsion system 1150 may be connected to the drivetrain of vehicle 1100, which may include a transmission, to allow propulsion of vehicle 1100. Propulsion system 1150 may be controlled in response to receiving a signal from throttle / accelerator 1152.

[0226] A steering system 1154, which may include a steering wheel, can be used to steer the vehicle 1100 (e.g., along a desired path or route) when the propulsion system 1150 is operating (e.g., when the vehicle is in motion). The steering system 1154 may receive signals from the steering actuator 1156. For fully automatic (level 5) functionality, the steering wheel may be optional.

[0227] The brake sensor system 1146 can be used to operate the vehicle brakes in response to receiving a flag from the brake actuator 1148 and / or the brake sensor.

[0228] It can include one or more System-on-Chip (SoC) 1104 ( Figure 11C One or more controllers 1136, including one or more GPUs, may provide (e.g., indicating commands) signals to one or more components and / or systems of vehicle 1100. For example, one or more controllers may send signals to operate vehicle brakes via one or more brake actuators 1148, to operate steering system 1154 via one or more steering actuators 1156, and to operate propulsion system 1150 via one or more throttles / accelerators 1152. One or more controllers 1136 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals indicating commands) to allow autonomous and / or human-assisted driving class drivers to operate vehicle 1100. One or more controllers 1136 may include a first controller 1136 for autonomous driving functions, a second controller 1136 for functional safety functions, a third controller 1136 for artificial intelligence functions (e.g., computer vision), a fourth controller 1136 for infotainment functions, a fifth controller 1136 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1136 may handle two or more of the above functions, two or more controllers 1136 may handle a single function, and / or any combination thereof.

[0229] One or more controllers 1136 may provide flags for controlling one or more components and / or systems of vehicle 1100 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example, but not limited to, Global Navigation Satellite System (“GNSS”) sensor 1158 (e.g., Global Positioning System sensor), RADAR sensor 1160, ultrasonic sensor 1162, LiDAR sensor 1164, Inertial Measurement Unit (IMU) sensor 1166 (e.g., accelerometer, gyroscope, magnetic compass, magnetometer, etc.), microphone 1196, stereo camera 1168, wide-angle camera 1170 (e.g., fisheye camera), infrared camera 1172, surround camera 1174 (e.g., 360-degree camera), long-range and / or medium-range camera 1198, speed sensor 1144 (e.g., for measuring the rate of vehicle 1100), vibration sensor 1142, steering sensor 1140, braking sensor 1146 (e.g., as part of braking sensor system 1146), and / or other sensor types.

[0230] One or more of the controllers 1136 may receive input (e.g., represented by input data) from the instrument panel 1132 of the vehicle 1100 and provide output (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 1134, an auditory sign, a speaker, and / or via other components of the vehicle 1100. These outputs may include information such as vehicle speed, rate, time, map data (e.g., [missing information]). Figure 11C Information such as the “HD” map 1122, location data (e.g., the location of vehicle 1100 on the map), direction, and the location of other vehicles (e.g., occupying a grid), as well as information about objects and their states perceived by controller 1136, etc. For example, HMI display 1134 may display information about the existence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, leaving 34B in two miles, etc.).

[0231] Vehicle 1100 also includes a network interface 1124, which can communicate via one or more networks using one or more wireless antennas 1126 and / or a modem. For example, network interface 1124 may be able to communicate via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multicarrier (“CDMA2000”), etc. One or more wireless antennas 1126 may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using one or more local area networks such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc., and / or one or more low-power wide area networks (LPWAN) such as LoRaWAN, SigFox, etc.

[0232] Figure 11B For use in accordance with some embodiments of this disclosure Figure 11A This is an example of the camera position and field of view of an example autonomous vehicle 1100. The camera and its respective field of view are an example embodiment and are not intended to be limiting. For example, additional and / or replaceable cameras may be included, and / or these cameras may be located at different positions on the vehicle 1100.

[0233] The camera type used for the camera may include, but is not limited to, a digital camera suitable for use with components and / or systems of vehicle 1100. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The camera may be able to use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a sharp-pixel camera, such as one having an RCCC, RCCB, and / or RBGC color filter array, may be used in efforts to improve light sensitivity.

[0234] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more of the cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0235] One or more of the cameras can be mounted in mounting components such as custom-designed (3D-printed) parts to cut off stray light and reflections from inside the vehicle (e.g., reflections from the dashboard in the windshield mirror) that may interfere with the camera's image data capture capabilities. Regarding wing mirror mounting components, the wing mirror components can be custom-3D printed so that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side-view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0236] A camera with a field of view that includes the environment in front of the vehicle 1100 (e.g., a front-facing camera) can be used for surround view to help identify forward paths and obstacles, and, with the assistance of one or more controllers 1136 and / or control SoCs, to provide information crucial for generating an occupancy grid and / or determining a preferred vehicle path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including lane departure warning (“LDW”), autonomous cruise control (“ACC”), and / or other functions such as traffic sign recognition.

[0237] A variety of cameras can be used in front-facing configurations, including, for example, monocular camera platforms that include CMOS (Complementary Metal-Oxide-Semiconductor) color imagers. Another example could be a wide-angle camera 1170, which can be used to perceive objects entering the field of view from the periphery (such as pedestrians, traffic at intersections, or bicycles). Although Figure 11B The image shows only one wide-angle camera, but any number of wide-angle cameras 1170 can be present on vehicle 1100. Furthermore, a remote camera 1198 (e.g., a pair of long-view stereo cameras) can be used for depth-based object detection, especially for objects for which a neural network has not yet been trained. The remote camera 1198 can also be used for object detection and classification, as well as basic object tracking.

[0238] One or more stereo cameras 1168 may also be included in a front-mounted configuration. One or more stereo cameras 1168 may include an integrated control unit comprising a scalable processing unit that can provide a multi-core microprocessor and programmable logic (“FPGA”) with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternative stereo cameras 1168 may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip capable of measuring the distance from the vehicle to a target object and using the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1168 may be used in addition to those described herein, or alternatively.

[0239] A camera (e.g., a side-view camera) having a field of view that includes the side of the vehicle 1100 can be used for surround view, providing information for creating and updating occupancy grids and generating side-impact collision warnings. For example, a surround camera 1174 (e.g., ...) Figure 11B The four surround cameras 1174 shown can be mounted on vehicle 1100. The surround cameras 1174 can include wide-angle cameras 1170, fisheye cameras, 360-degree cameras, and / or similar devices. Four examples are provided; the four fisheye cameras can be positioned at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 1174 (e.g., left, right, and rear) and can utilize one or more other cameras (e.g., forward-facing cameras) as a fourth surround-view camera.

[0240] A camera (e.g., a rear-view camera) having a field of view that includes the environment behind the vehicle 1100 can be used for parking assistance, surround view, rear collision warning, and creating and updating occupancy grids. A wide variety of cameras can be used, including but not limited to those also suitable as front-facing cameras as described herein (e.g., long-range and / or mid-range camera 1198, stereo camera 1168, infrared camera 1172, etc.).

[0241] Figure 11C For use in accordance with some embodiments of this disclosure Figure 11AThe example autonomous vehicle 1100 is illustrated in the block diagram of an example system architecture. It should be understood that this arrangement, and other arrangements described herein, are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities, which may be implemented as discrete or distributed components or in combination with other components, and in any suitable combination and location. The various functions described herein as being performed by these entities can be implemented via hardware, firmware, and / or software. For example, the various functions can be implemented by a processor executing instructions stored in memory.

[0242] Figure 11C Each component, feature, and system in vehicle 1100 is illustrated as being connected via bus 1102. Bus 1102 may include a Controller Area Network (CAN) data interface (or, alternatively, referred to herein as the "CAN bus"). CAN may be a network within vehicle 1100 used to assist in controlling various features and functions of vehicle 1100, such as braking, acceleration, steering, windshield wipers, and other driving functions. CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button positions, and / or other vehicle status indicators. CAN bus may be ASIL B compliant.

[0243] Although bus 1102 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet may be used in addition to or alternatively to a CAN bus. Furthermore, although bus 1102 is represented by a single line, this is not intended to be limiting. For example, any number of buses 1102 may exist, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 1102 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 1102 may be used for collision avoidance functions, and a second bus 1102 may be used for driving control. In any example, each bus 1102 may communicate with any component of vehicle 1100, and two or more buses 1102 may communicate with the same component. In some examples, each SoC 1004, each controller 1136, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors in the vehicle 1100) and may be connected to a common bus such as the CAN bus.

[0244] Vehicle 1100 may include one or more controllers 1136, such as those described herein. Figure 11A The controllers described herein. Controller 1136 can be used for a wide variety of functions. Controller 1136 can be coupled to any other different components and systems of vehicle 1100 and can be used for the control of vehicle 1100, artificial intelligence of vehicle 1100, infotainment and / or similar functions of vehicle 1100.

[0245] Vehicle 1100 may include one or more System-on-Chip (SoC) 1104. SoC 1104 may include CPU 1106, GPU 1108, processor 1110, cache 1112, accelerator 1114, data storage 1116, and / or other components and features not shown. SoC 1104 can be used to control vehicle 1100 across a wide variety of platforms and systems. For example, one or more SoCs 1104 may be combined with an HD map 1122 in a system (e.g., the system of vehicle 1100), the HD map being accessible from one or more servers (e.g., via a network interface 1124). Figure 11D One or more servers (1178) receive map refresh and / or updates.

[0246] CPU 1106 may include CPU clusters or CPU complexes (or, alternatively, referred to herein as "CCPLEX"). CPU 1106 may include multiple cores and / or L2 cache. For example, in some embodiments, CPU 1106 may include eight cores in a coherent multiprocessor configuration. In some embodiments, CPU 1106 may include four dual-core clusters, each cluster having a dedicated L2 cache (e.g., 2MB L2 cache). CPU 1106 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, such that any combination of clusters of CPU 1106 can be active at any given time.

[0247] CPU 1106 can implement power management capabilities including one or more of the following features: automatic clock gating of hardware blocks when idle to save dynamic power; clock gating of each core when the core is not actively executing instructions due to the execution of WFI / WFE instructions; independent power gating of each core; independent clock gating of each core cluster when all cores are clock-gated or power-gated; and / or independent power gating of each core cluster when all cores are power-gated. CPU 1106 can further implement enhanced algorithms for managing power states, wherein allowed power states and desired wake-up times are specified, and the hardware / microcode determines the optimal power state to enter for the core, cluster, and CCPLEX. The processing core can support simplified power state entry sequences in software, with this work offloaded to the microcode.

[0248] GPU 1108 may include an integrated GPU (or, alternatively, referred to herein as an "iGPU"). GPU 1108 may be programmable and efficient for parallel workloads. In some examples, GPU 1108 may use an enhanced tensor instruction set. GPU 1108 may include one or more streaming microprocessors, wherein each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96KB of storage capacity), and two or more of these streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage capacity). In some embodiments, GPU 1108 may include at least eight streaming microprocessors. GPU 1108 may use a computation application programming interface (API). Furthermore, GPU 1108 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0249] In automotive and embedded applications, the GPU 1108 can be power-optimized for optimal performance. For example, the GPU 1108 can be fabricated on FinFETs. However, this is not intended to be limiting, and the GPU 1108 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can combine several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to leverage the mixture of computation and addressing computations to provide efficient execution of workloads. Streaming microprocessors may include independent thread scheduling capabilities to allow for finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors may include combined L1 data caches and shared memory units to improve performance while simplifying programming.

[0250] The GPU 1108 may include, in some examples, a High Bandwidth Memory (HBM) and / or a 16GB HBM2 memory subsystem providing a peak memory bandwidth of approximately 900GB / s. In some examples, in addition to HBM memory or alternatively, Synchronous Graphics Random Access Memory (SGRAM), such as Generation 5 Graphics Double Data Rate Synchronous Random Access Memory (GDDR5), may be used.

[0251] The GPU 1108 may include unified memory technology, which includes access counters to allow memory pages to be migrated more precisely to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, Address Translation Service (ATS) support can be used to allow the GPU 1108 to directly access the CPU 1106 page tables. In such examples, when the GPU 1108 Memory Management Unit (MMU) experiences a miss, the address translation request can be transferred to the CPU 1106. In response, the CPU 1106 can look up the virtual-physical mapping for the address in its page tables and transfer the translation back to the GPU 1108. Thus, unified memory technology can allow a single unified virtual address space for the memory of both the CPU 1106 and the GPU 1108, simplifying GPU 1108 programming and porting applications to the GPU 1108.

[0252] In addition, the GPU 1108 may include access counters that track how frequently the GPU 1108 accesses the memory of other processors. Access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses those pages most frequently.

[0253] SoC 1104 may include any number of caches 1112, including those described herein. For example, cache 1112 may include an L3 cache available to both CPU 1106 and GPU 1108 (e.g., it is connected to both CPU 1106 and GPU 1108). Cache 1112 may include a write-back cache, which can track the state of rows, for example, using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache may include 4 MB or more, but a smaller cache size may also be used.

[0254] SoC 1104 may include an arithmetic logic unit (ALU) that can be utilized in processing of any of the various tasks or operations performed on vehicle 1100, such as processing a DNN. Furthermore, SoC 1104 may include a floating-point unit (FPU) (or other mathematical coprocessor or digital coprocessor type) for performing mathematical operations within the system. For example, SoC 1104 may include one or more FPUs integrated as execution units within CPU 1106 and / or GPU 1108.

[0255] SoC 1104 may include one or more accelerators 1114 (e.g., hardware accelerators, software accelerators, or combinations thereof). For example, SoC 1104 may include a hardware accelerator cluster, which may include optimized hardware accelerators and / or large on-chip memory. This large on-chip memory (e.g., 4MB SRAM) can enable the hardware accelerator cluster to accelerate neural networks and other computations. The hardware accelerator cluster can be used to secondary GPU 1108 and offload some tasks from GPU 1108 (e.g., freeing up more cycles of GPU 1108 to perform other tasks). As an example, accelerator 1114 can be used for targeted workloads (e.g., perceptrons, convolutional neural networks (CNNs), etc.) that are stable enough to be easily controlled for acceleration. When used herein, the term "CNN" can include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0256] Accelerator 1114 (e.g., a hardware accelerator cluster) may include a Deep Learning Accelerator (DLA). The DLA may include one or more Tensor Processing Units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may be an accelerator configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and optimized for performing image processing functions. The DLA may be further optimized for a specific set of neural network types and floating-point operations and inference. The DLA is designed to provide higher performance per millimeter than a general-purpose GPU and significantly outperform CPUs. The TPU can perform several functions, including single-instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, and post-processor functions.

[0257] DLA can execute neural networks, especially CNNs, quickly and efficiently on processed or unprocessed data for any function across a wide variety of applications, such as, but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and recognition using data from microphones; CNNs for face recognition and vehicle owner recognition using data from camera sensors; and / or CNNs for safety and / or safety-related events.

[0258] The DLA can perform any function of the GPU 1108, and by using inference accelerators, for example, designers can make the DLA or GPU 1108 target any function. For example, designers can focus the CNN processing and floating-point operations on the DLA and leave other functions to the GPU 1108 and / or other accelerators 1114.

[0259] Accelerator 1114 (e.g., a hardware accelerator cluster) may include a programmable vision accelerator (PVA), which may alternatively be referred to herein as a computer vision accelerator. The PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA may include, for example, but not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0260] RISC cores can interact with image sensors (such as the image sensor of any camera described herein), image label processors, and / or similar objects. Each of these RISC cores may include any amount of memory. Depending on the embodiment, the RISC core may use any of several protocols. In some examples, the RISC core may execute a real-time operating system (RTOS). RISC cores may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0261] DMA enables PVA components to access system memory independently of the CPU 1106. DMA can support any number of features to provide optimizations to the PVA, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which can include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0262] A vector processor can be a programmable processor designed to efficiently and flexibly execute programming for computer vision algorithms and provide tag processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital tag processor, such as, for example, a Single Instruction Multiple Data (SIMD) or Very Long Instruction Word (VLIW) digital tag processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0263] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. As a result, in some examples, each of the vector processors may be configured to execute independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even different algorithms on a sequence of images or portions of an image. Among other things, any number of PVAs may be included in a cluster of hardware accelerators, and any number of vector processors may be included in each of these PVAs. Furthermore, the PVA may include additional error correction code (ECC) memory to enhance overall system security.

[0264] Accelerator 1114 (e.g., a hardware accelerator cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for accelerator 1114. In some examples, on-chip memory may include at least 4MB of SRAM consisting of, for example, but not limited to, eight field-configurable memory blocks, accessible by both the PVA and DLA. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA may access memory via a backbone that provides high-speed memory access to the PVA and DLA. The backbone may include (e.g., using an APB) an on-chip computer vision network that interconnects the PVA and DLA to memory.

[0265] On-chip computer vision networks can include an interface that determines whether both the PVA and DLA provide a ready and valid flag before transmitting any control flags / addresses / data. Such an interface can provide separate phases and channels for transmitting control flags / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 615010 standards, but other standards and protocols can also be used.

[0266] In some examples, SoC 1104 may include, for example, a real-time ray tracing hardware accelerator as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. This real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) to generate real-time visualization simulations for RADAR sign interpretation, sound propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison with LiDAR data for localization and / or other functional purposes, and / or for other uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations.

[0267] Accelerators 1114 (e.g., hardware accelerator clusters) have broad applications in autonomous driving. PVAs can be programmable vision accelerators used in critical processing stages of ADAS and autonomous vehicles. The capabilities of PVAs are a good match for algorithmic domains requiring predictable processing, low power, and low latency. In other words, PVAs perform well in semi-dense or dense rule computation, even on small datasets requiring predictable runtimes with low latency and low power. Therefore, in the context of platforms for autonomous vehicles, PVAs are designed to run classical computer vision algorithms because they are effective in object detection and integer mathematical operations.

[0268] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. In some examples, semi-global matching-based algorithms may be used, but this is not intended to be limiting. Many applications for Level 3-5 autonomous driving require instantaneous motion estimation / stereo matching (e.g., from moving structures, pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions on input from two monocular cameras.

[0269] In some examples, PVA can be used to perform intensive optical flow, providing processed RADAR data from the raw RADAR data (e.g., using 4D Fast Fourier Transform). In other examples, PVA is used for time-of-flight depth processing, which, for example, involves processing raw time-of-flight data to provide processed time-of-flight data.

[0270] DLA can be used to run any type of network to enhance control and driving safety, including, for example, neural networks that output a confidence metric for each object detection. Such a confidence value can be interpreted as a probability or as providing a relative “weight” for each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a threshold for the confidence and only consider detections exceeding the threshold as true positives. In an Automatic Emergency Braking (AEB) system, false positives can cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run neural networks to regress the confidence values. The neural network can take at least some subset of parameters as its input, such as bounding box dimensions, ground plane estimates obtained (e.g. from another subsystem), outputs from inertial measurement unit (IMU) sensors 1166 related to the orientation and distance of vehicle 1100, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LiDAR sensor 1164 or RADAR sensor 1160), etc.

[0271] SoC 1104 may include one or more data stores 1116 (e.g., memory). The data store 1116 may be on-chip memory of SoC 1104, which may store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and security, the data store 1116 may be large enough to store multiple instances of the neural network. The data store 1116 may include L2 or L3 cache 1112. References to the data store 1116 may include references to memory associated with the PVA, DLA, and / or other accelerators 1114 as described herein.

[0272] SoC 1104 may include one or more processors 1110 (e.g., embedded processors). Processor 1110 may include a startup and power management processor, which may be a dedicated processor and subsystem for handling startup power and management functions, as well as safety implementation. The startup and power management processor may be part of the SoC 1104 startup sequence and may provide runtime power management services. The startup power and management processor may provide clock and voltage programming, auxiliary system low-power state transitions, SoC 1104 thermal and temperature sensor management, and / or SoC 1104 power state management. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and SoC 1104 may use the ring oscillator to detect the temperature of CPU 1106, GPU 1108, and / or accelerator 1114. If it is determined that the temperature exceeds a threshold, the startup and power management processor may enter a temperature fault routine and place SoC 1104 into a lower power state and / or place vehicle 1100 into a driver-safe parking mode (e.g., safely stop vehicle 1100).

[0273] The processor 1110 may also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio via multiple interfaces, as well as a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital flag processor and dedicated RAM.

[0274] The processor 1110 may also include an always-on-processor engine that can provide the necessary hardware features to support low-power sensor management and wake-up use cases. This always-on-processor engine may include a processor core, tightly coupled RAM, support for peripherals (such as timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0275] The processor 1110 may also include a security cluster engine, which includes a dedicated processor subsystem for handling security management for automotive applications. The security cluster engine may include two or more processor cores, tightly coupled RAM, support for peripheral devices (e.g., timers, interrupt controllers, etc.), and / or routing logic. In secure mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic that detects any differences between their operations.

[0276] The processor 1110 may also include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0277] The processor 1110 may also include a high dynamic range flag processor, which may include an image flag processor, which is a hardware engine that is part of the camera processing pipeline.

[0278] Processor 1110 may include a video image compositer, which may be (e.g., implemented on a microprocessor) a processing block, implementing video post-processing functions required by the video playback application to generate the final image for the player window. The video image compositer may perform lens distortion correction on the wide-angle camera 1170, the surround camera 1174, and / or the in-cabin monitoring camera sensor. The in-cabin monitoring camera sensor is preferably monitored by a neural network running on another instance of an advanced SoC, configured to recognize in-cabin events and respond accordingly. The in-cabin system may perform lip reading to activate mobile phone services and make calls, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. Some functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled in other situations.

[0279] Video image compositers can include enhanced temporal denoising for both spatial and temporal noise reduction. For example, in the case of motion in the video, denoising appropriately weights spatial information, reducing the weight of information provided by neighboring frames. In cases where the image or part of the image does not contain motion, the temporal denoising performed by the video image compositer can use information from previous images to reduce noise in the current image.

[0280] The video image compositer can also be configured to perform stereo correction on input stereo lens frames. When the operating system desktop is in use and the GPU 1108 does not need to continuously render new surfaces, the video image compositer can be further used for user interface components. Even when the GPU 1108 is powered on and active, performing 3D rendering, the video image compositer can be used to offload the GPU 1108 to improve performance and responsiveness.

[0281] The SoC 1104 may also include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and pixel-related input functions for receiving video and input from a camera. The SoC 1104 may also include an input / output controller that can be software-controlled and can be used to receive I / O flags not submitted to a specific role.

[0282] SoC 1104 may also include a wide range of peripheral interfaces to allow communication with peripherals, audio codecs, power management and / or other devices. SoC 1104 can be used to process data from cameras and sensors (e.g., LiDAR sensor 1164, RADAR sensor 1160, etc., which can be connected via Gigabit Multimedia Serial Link and Ethernet), data from bus 1102 (e.g., vehicle 1100 speed, steering wheel position, etc.), and data from GNSS sensor 1158 (connected via Ethernet or CAN bus). SoC 1104 may also include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine and can be used to free up CPU 1106 from routine data management tasks.

[0283] The SoC 1104 can be an end-to-end platform with a flexible architecture spanning Automation Levels 3-5, providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS technologies for diversity and redundancy, along with deep learning tools to deliver a flexible and reliable driving software stack. The SoC 1104 can be faster, more reliable, and even more energy- and space-efficient than conventional systems. For example, when combined with the CPU 1106, GPU 1108, and data storage 1116, the accelerator 1114 can provide a fast and efficient platform for Level 3-5 autonomous vehicles.

[0284] Therefore, this technology offers capabilities and functionalities that cannot be achieved through conventional systems. For example, computer vision algorithms can be executed on CPUs, which can be configured using high-level programming languages ​​such as C to execute a wide variety of processing algorithms across a diverse range of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for automotive ADAS applications and practical Level 3-5 autonomous vehicles.

[0285] In contrast to conventional systems, the techniques described in this paper, by providing CPU complexes, GPU complexes, and hardware accelerator clusters, allow multiple neural networks to be executed simultaneously and / or sequentially, and the results combined to achieve Level 3–5 autonomous driving capabilities. For example, a CNN executed on a DLA or dGPU (e.g., GPU 1120) could include text and word recognition, allowing a supercomputer to read and understand traffic signs, including those for which neural networks have not yet been specifically trained. The DLA could also include a neural network capable of recognizing, interpreting, and providing semantic understanding of the signs, and passing that semantic understanding to a path planning module running on the CPU complex.

[0286] As another example, multiple neural networks can operate simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of "Caution: Flashing lights indicate icy conditions," along with a light, can be interpreted independently or jointly by several neural networks. The sign itself can be recognized as a traffic sign by a deployed first neural network (e.g., a trained neural network), and the text "Flashing lights indicate icy conditions" can be interpreted by a deployed second neural network that informs the vehicle's path planning software (preferably executing on a CPU complex) that icy conditions exist when the flashing lights are detected. The flashing lights can be identified by a deployed third neural network operating across multiple frames, informing the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks can operate simultaneously, for example, within a DLA and / or on a GPU 1108.

[0287] In some examples, the CNN used for facial recognition and owner identification can use data from camera sensors to identify the presence of an authorized driver and / or owner of vehicle 1100. A processing engine always on the sensors can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in safe mode, to disable the vehicle when the owner leaves. In this way, SoC 1104 provides security against theft and / or carjacking.

[0288] In another example, the CNN used for emergency vehicle detection and identification can use data from microphone 1196 to detect and identify emergency vehicle siren. In contrast to conventional systems that use a general classifier to detect siren and manually extract features, SoC 1104 uses a CNN to classify environmental and urban sounds as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative shut-off rate of emergency vehicles (e.g., by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the localized area in which the vehicle operates, as identified by GNSS sensor 1158. Thus, for example, when operating in the EU, the CNN will seek to detect EU siren, and when operating in the US, the CNN will seek to identify siren only in North America. Once an emergency vehicle is detected, with the assistance of ultrasonic sensor 1162, the control program can be used to execute emergency vehicle safety routines, causing the vehicle to slow down, pull over to the side of the road, stop, and / or idle until the emergency vehicle passes.

[0289] The vehicle may include a CPU 1118 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 1104 via a high-speed interconnect (e.g., PCIe). The CPU 1118 may include, for example, an X106 processor. The CPU 1118 can be used to perform any of a wide variety of functions, including, for example, arbitrating the results of potential inconsistencies between ADAS sensors and the SoC 1104, and / or monitoring the status and health of the controller 1136 and / or the infotainment SoC 1130.

[0290] Vehicle 1100 may include a GPU 1120 (e.g., a discrete GPU or dGPU) that can be coupled to SoC 1104 via a high-speed interconnect (e.g., NVIDIA's NVLINK). GPU 1120 may provide additional artificial intelligence capabilities, for example by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based at least in part on inputs (e.g., sensor data) from sensors of vehicle 1100.

[0291] Vehicle 1100 may also include a network interface 1124, which may include one or more wireless antennas 1126 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). Network interface 1124 can be used to enable wireless connectivity via the Internet to the cloud (e.g., with server 1178 and / or other network devices), with other vehicles, and / or with computing devices (e.g., passenger client devices). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across networks and via the Internet). A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide vehicle 1100 with information about vehicles approaching vehicle 1100 (e.g., vehicles in front, to the side, and / or behind vehicle 1100). This functionality may be part of vehicle 1100's cooperative adaptive cruise control function.

[0292] Network interface 1124 may include a SoC that provides modulation and demodulation functions and enables controller 1136 to communicate via a wireless network. Network interface 1124 may include an RF front-end for up-conversion from baseband to RF and down-conversion from RF to baseband. Frequency conversion can be performed using known processes and / or using a superheterodyne process. In some examples, the RF front-end functionality may be provided by a separate chip. The network interface may include wireless functions for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0293] Vehicle 1100 may also include data storage 1128, which may include off-chip (e.g., off-chip SoC 1104) storage devices. Data storage 1128 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disk, and / or other components and / or devices capable of storing at least one bit of data. Storage 1128

[0294] Vehicle 1100 may also include a GNSS sensor 1158. The GNSS sensor 1158 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used for auxiliary mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 1158 can be used, including, for example, but not limited to, GPS using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0295] Vehicle 1100 may also include a RADAR sensor 1160. The RADAR sensor 1160 can be used by vehicle 1100 for remote vehicle detection even in dark and / or inclement weather conditions. The RADAR functional safety level may be ASIL B. The RADAR sensor 1160 can use CAN and / or bus 1102 (e.g., to transmit data generated by the RADAR sensor 1160) for control and access to object tracking data, and in some examples, Ethernet access for accessing raw data. A wide variety of RADAR sensor types can be used. For example, and without limitation, the RADAR sensor 1160 can be adapted for front, rear, and side RADAR use. In some examples, a pulse Doppler RADAR sensor is used.

[0296] RADAR sensor 1160 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In some examples, long-range RADAR can be used for adaptive cruise control functions. A long-range RADAR system can provide a wide field of view (e.g., within 250m) achieved through two or more independent scans. RADAR sensor 1160 can help distinguish between stationary and moving objects and can be used by ADAS systems for emergency braking assist and forward collision warning. Long-range RADAR sensors can include a single-site multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record the vehicle 1100's surroundings at higher rates with minimal traffic interference from adjacent lanes. The other two antennas can extend the field of view, enabling rapid detection of vehicles entering or leaving the vehicle 1100's lane.

[0297] As an example, a mid-range RADAR system can include a range of up to 160m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 150 degrees (rear). Short-range RADAR systems can include, but are not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor blind spots behind and beside the vehicle.

[0298] Short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0299] Vehicle 1100 may also include ultrasonic sensors 1162. Ultrasonic sensors 1162, which may be positioned at the front, rear, and / or sides of vehicle 1100, can be used for parking assistance and / or creating and updating occupancy grids. A wide variety of ultrasonic sensors 1162 can be used, and different ultrasonic sensors 1162 can be used for different detection ranges (e.g., 2.5m, 4m). Ultrasonic sensors 1162 can operate at functional safety level ASIL B.

[0300] Vehicle 1100 may include a LiDAR sensor 1164. The LiDAR sensor 1164 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LiDAR sensor 1164 may be of functional safety level ASIL B. In some examples, vehicle 1100 may include multiple LiDAR sensors 1164 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0301] In some examples, the LiDAR sensor 1164 may be able to provide a list of objects and their distances within a 360-degree field of view. Commercially available LiDAR sensors 1164 may have an advertising range of, for example, approximately 100m, with an accuracy of 2cm-3cm, and support for 100Mbps Ethernet connectivity. In some examples, one or more non-protruding LiDAR sensors 1164 may be used. In such examples, the LiDAR sensor 1164 may be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of vehicle 1100. In such examples, the LiDAR sensor 1164 may provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, even for low-reflectivity objects, with a range of 200m. Front-mounted LiDAR sensors 1164 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0302] In some examples, LiDAR technologies such as 3D flash LiDAR can also be used. 3D flash LiDAR uses a flash of laser light as the emission source to illuminate the vehicle's surroundings up to approximately 200 meters. The flash LiDAR unit includes a receiver that records the laser pulse propagation time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LiDAR allows for the generation of highly accurate and distortion-free images of the surrounding environment using each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of the vehicle 1100. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras (e.g., non-scanning LiDAR devices) without moving parts other than a fan. Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-registered intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 1164 is less susceptible to motion blur, vibration, and / or shock.

[0303] The vehicle may also include an IMU sensor 1166. In some examples, the IMU sensor 1166 may be located at the center of the rear axle of the vehicle 1100. The IMU sensor 1166 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 1166 may include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 1166 may include an accelerometer, a gyroscope, and a magnetometer.

[0304] In some embodiments, the IMU sensor 1166 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 1166 can enable the vehicle 1100 to estimate heading by directly observing and correlating velocity changes from GPS to the IMU sensor 1166 without requiring input from a magnetic sensor. In some examples, the IMU sensor 1166 and the GNSS sensor 1158 can be combined into a single integrated unit.

[0305] The vehicle may include a microphone 1196 placed in and / or around the vehicle 1100. Among other things, the microphone 1196 may be used for emergency vehicle detection and identification.

[0306] The vehicle may also include any number of camera types, including stereo camera 1168, wide-angle camera 1170, infrared camera 1172, surround camera 1174, long-range and / or mid-range camera 1198, and / or other camera types. These cameras can be used to capture image data around the entire perimeter of the vehicle 1100. The types of cameras used depend on the embodiment and the requirements of the vehicle 1100, and any combination of camera types can be used to provide the necessary coverage around the vehicle 1100. Furthermore, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As an example and without limitation, these cameras may support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described herein with respect to... Figure 11A and Figure 11B It was described in more detail.

[0307] Vehicle 1100 may also include vibration sensor 1142. Vibration sensor 1142 can measure vibrations of vehicle components such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 1142 are used, differences between vibrations can be used to determine friction or slippage on the road surface (e.g., when there is a vibration difference between the powered drive shaft and the free-rotating shaft).

[0308] Vehicle 1100 may include ADAS system 1138. In some examples, ADAS system 1138 may include SoC. ADAS system 1138 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic warning (RCTW), collision warning system (CWS), lane centering (LC) and / or other features and functions.

[0309] The ACC system can use a RADAR sensor 1160, a LiDAR sensor 1164, and / or a camera. The ACC system may include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to vehicles immediately in front of vehicle 1100 and automatically adjusts the vehicle speed to maintain a safe distance. Lateral ACC performs distance holding and, if necessary, advises vehicle 1100 to change lanes. Lateral ACC is associated with other ADAS applications such as LCA and CWS.

[0310] CACC uses information from other vehicles, which can be received indirectly from other vehicles via a wireless link or through a network connection (e.g., via the Internet) through network interface 1124 and / or wireless antenna 1126. Direct links can be provided by vehicle-to-vehicle (V2V) communication links, while indirect links can be infrastructure-to-vehicle (I2V) communication links. Typically, the V2V communication concept provides information about vehicles immediately ahead (e.g., vehicles immediately in front of vehicle 1100 and in the same lane), while the I2V communication concept provides information about traffic further ahead. A CACC system can include either or both of these I2V and V2V information sources. Given information about vehicles ahead of vehicle 1100, CACC can be more reliable, and it has the potential to improve traffic flow and reduce road congestion.

[0311] The Forward-Looking Warning (FCW) system is designed to alert the driver to hazards, enabling the driver to take corrective action. The FCW system uses a front-facing camera and / or RADAR sensor 1160 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components. The FCW system can provide warnings in the form of, for example, audible, visual, haptic, and / or rapid braking pulses.

[0312] An AEB (Autonomous Emergency Braking) system detects an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system can use a front-facing camera and / or RADAR sensor 1160 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes to attempt to prevent or at least mitigate the effects of the predicted collision. The AEB system may include technologies such as dynamic brake support and / or collision approach braking.

[0313] The Lane Departure Warning (LDW) system provides visual, auditory, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle crosses lane markings. The LDW system is deactivated when the driver indicates intentional lane departure by activating the turn sign. The LDW system can utilize a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as displays, speakers, and / or vibrating components.

[0314] The Lane Keeping Assist (LKA) system is a variant of the Lane Departure Warning (LDW) system. If vehicle 1100 begins to leave its lane, the LKA system provides corrective steering input or braking to vehicle 1100. The Blind Spot Warning (BSW) system detects and warns the driver of vehicles in the vehicle's blind spot. The BSW system can provide visual, audible, and / or tactile warnings to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses a turn signal. The BSW system can use a rear-view camera and / or a RADAR sensor 1160.

[0315] The RCTW system can provide visual, auditory, and / or tactile notifications when an object is detected outside the range of the rear camera while the vehicle 1100 is reversing. Some RCTW systems include AEB to ensure the application of vehicle braking to avoid a collision. The RCTW system may use one or more rear-mounted RADAR sensors 1160 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as displays, speakers, and / or vibrating components.

[0316] Conventional ADAS systems can be prone to false positives, which can be annoying and distracting for the driver, but typically not catastrophic, as they alert the driver and allow them to determine whether a safe condition truly exists and take appropriate action. However, in the autonomous vehicle 1100, in the event of conflicting results, the vehicle 1100 itself must decide whether to heed the results from the main computer or auxiliary computer (e.g., the first controller 1136 or the second controller 1136). For example, in some embodiments, the ADAS system 1138 may be a backup and / or auxiliary computer for providing perception information to a backup computer rationality module. The backup computer rationality monitor may run redundant and varied software on hardware components to detect faults in perception and dynamic driving tasks. Outputs from the ADAS system 1138 may be provided to a supervisory MCU. If the outputs from the main computer and the auxiliary computer conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0317] In some examples, the master computer can be configured to provide a confidence score to the supervisory MCU, indicating the master computer's confidence level in the selected result. If the confidence score exceeds a threshold, the supervisory MCU can follow the master computer's direction regardless of whether the auxiliary computer provides conflicting or inconsistent results. If the confidence score does not meet the threshold and the master and auxiliary computers indicate different results (e.g., conflict), the supervisory MCU can arbitrate between these computers to determine the appropriate result.

[0318] The supervisory MCU can be configured to run a neural network trained and configured to determine, at least in part, the conditions under which the auxiliary computer provides a false alarm, based on outputs from both the host and auxiliary computers. Thus, the neural network in the supervisory MCU can learn when the output of the auxiliary computer can be trusted and when it cannot. For example, when the auxiliary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying a metallic object that is not actually dangerous, such as a drain grid or manhole cover that triggers an alarm. Similarly, when the auxiliary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU may include at least one of a DLA or GPU suitable for running the neural network using associated memory. In a preferred embodiment, the supervisory MCU may include a component of SoC 1104 and / or be included as a component of SoC 1104.

[0319] In other examples, ADAS system 1138 may include an auxiliary computer that performs ADAS functions using conventional computer vision rules. This allows the auxiliary computer to use classic computer vision rules (if-then), and the presence of neural networks in the supervising MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identity make the entire system more fault-tolerant, especially for failures caused by software (or software-hardware interface) functionality. For instance, if a software vulnerability or bug exists in the software running on the host computer and non-identical software code running on the auxiliary computer provides the same overall result, the supervising MCU can be more confident that the overall result is correct and that the vulnerability in the software or hardware on the host computer does not cause a substantial error.

[0320] In some examples, the output of ADAS system 1138 can be fed to the perception block and / or the dynamic driving task block of the main computer. For example, if ADAS system 1138 issues a forward collision warning because an object is immediately in front, the perception block can use this information when identifying the object. In other examples, the assistance computer can have its own neural network, which is trained and thus reduces the risk of false positives as described herein.

[0321] Vehicle 1100 may also include an infotainment SoC 1130 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 1130 may include a combination of hardware and software that can be used to provide vehicle 1100 with audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.) and / or information services (e.g., navigation systems, rear parking assistance, radio data systems, vehicle-related information such as fuel level, total coverage distance, brake fuel level, fuel level, door opening / closing, air filter information, etc.). For example, the infotainment SoC 1130 may include a radio, disc player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, WiFi, steering wheel audio controls, hands-free voice controls, head-up display (HUD), HMI display 1134, telematics device, control panel (e.g., for controlling and / or interacting with various components, features, and / or systems) and / or other components. The infotainment SoC 1130 may further be used to provide information (e.g., visual and / or auditory) to users of the vehicle, such as information from the ADAS system 1138, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0322] The infotainment SoC 1130 may include GPU functionality. The infotainment SoC 1130 can communicate with other devices, systems, and / or components of the vehicle 1100 via bus 1102 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 1130 may be coupled to a supervisory MCU, enabling the GPU of the infotainment system to perform some autonomous driving functions in the event of a failure of the main controller 1136 (e.g., the primary and / or backup computer of the vehicle 1100). In such an example, the infotainment SoC 1130 may place the vehicle 1100 into a driver-safe parking mode as described herein.

[0323] Vehicle 1100 may also include instrument panel 1132 (e.g., digital instrument cluster, electronic instrument cluster, digital instrument panel, etc.). Instrument panel 1132 may include a controller and / or supercomputer (e.g., a discrete controller or supercomputer). Instrument panel 1132 may include a set of instruments such as speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between infotainment SoC 1130 and instrument panel 1132. In other words, instrument panel 1132 may be included as part of infotainment SoC 1130, or vice versa.

[0324] Figure 11D For cloud-based servers and according to some embodiments of this disclosure Figure 11A This is a schematic diagram of a system for communication between example autonomous vehicles 1100. System 1076 may include server 1178, network 1090, and vehicles including vehicle 1100. Server 1178 may include multiple GPUs 1084(A)-1084(H) (collectively referred to herein as GPU 1084), PCIe switches 1082(A)-1082(D) (collectively referred to herein as PCIe switch 1082), and / or CPUs 1080(A)-1080(B) (collectively referred to herein as CPU 1080). GPU 1084, CPU 1080, and PCIe switches may be interconnected with high-speed interconnects and / or PCIe connections 1086, such as, but not limited to, NVLink interface 1088 developed by NVIDIA. In some examples, GPU 1084 is connected via NVLink and / or NVSwitch SoC, and GPU 1084 and PCIe switch 1082 are connected via PCIe interconnect. Although the diagram illustrates eight GPUs 1084, two CPUs 1080, and two PCIe switches, it is not intended to be limiting. Depending on the embodiment, each of the servers 1178 may include any number of GPUs 1084, CPUs 1080, and / or PCIe switches. For example, each of the servers 1178 may include eight, sixteen, thirty-two, and / or more GPUs 1084.

[0325] Server 1178 can receive image data from vehicles via network 1090, representing images of unexpected or altered road conditions, such as recently commenced roadworks. Server 1178 can also transmit neural network 1092, updated neural network 1092, and / or map information 1094, including information about traffic and road conditions, to vehicles via network 1090. Updates to map information 1094 may include updates to HD map 1122, such as information about construction sites, potholes, bends, floods, or other obstacles. In some examples, neural network 1092, updated neural network 1092, and / or map information 1094 may have been generated from new training and / or data received from any number of vehicles in the environment, and / or based on experience gained from training performed at a data center (e.g., using server 1178 and / or other servers).

[0326] Server 1178 can be used to train machine learning models (e.g., neural networks) based on training data. Training data can be generated by the vehicle and / or generated in a simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., where the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not labeled and / or preprocessed (e.g., where the neural network does not require supervised learning). Training can be performed according to any one or more categories of machine learning techniques, including but not limited to categories such as: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, it can be used by the vehicle (e.g., transmitted to the vehicle via network 1090), and / or the machine learning model can be used by server 1178 to remotely monitor the vehicle.

[0327] In some examples, server 1178 can receive data from vehicles and apply that data to state-of-the-art real-time neural networks for real-time intelligent inference. Server 1178 may include a deep learning supercomputer powered by GPU 1084 and / or a dedicated AI computer, such as the DGX and DGX station machines developed by NVIDIA. However, in some examples, server 1178 may include a deep learning infrastructure in a data center that uses only CPU power.

[0328] The deep learning infrastructure of server 1178 may be capable of rapid real-time inference and can be used to assess and verify the health status of the processor, software, and / or associated hardware in vehicle 1100. For example, the deep learning infrastructure may receive periodic updates from vehicle 1100, such as image sequences and / or objects located in those image sequences that vehicle 1100 has already located (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure may run its own neural network to identify objects and compare them with objects identified by vehicle 1100. If the results do not match and the infrastructure concludes that the AI ​​in vehicle 1100 has malfunctioned, then server 1178 may transmit a flag to vehicle 1100, instructing vehicle 1100's fail-safe computer to take control, notify passengers, and complete a safe stopping operation.

[0329] For inference, server 1178 may include GPU 1084 and one or more programmable inference accelerators (such as NVIDIA's TensorRT). The combination of a GPU-powered server and inference acceleration enables real-time response. In other examples, such as where performance is less critical, CPU, FPGA, and other processor-powered servers can be used for inference.

[0330] This disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, which are executed by a computer or other machine such as a personal digital assistant or other handheld device. Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a wide variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. This disclosure can also be practiced in distributed computing environments in which tasks are performed by remote processing devices linked via a communication network.

[0331] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0332] This document describes in detail the subject matter of this disclosure to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the discloser has envisioned that the claimed subject matter may be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of the method employed, these terms should not be construed as suggesting any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

[0333] For example, the subject matter of this disclosure is illustrated by various aspects described below. For convenience, various examples of aspects of this disclosure are described as numbered examples (1, 2, 3, etc.). These are provided by way of example only and do not limit this disclosure. Unless the context otherwise indicates, aspects of the various implementations described herein may be omitted, replaced with aspects of other implementations, or combined with aspects of other implementations. For example, one or more aspects of Example 1 below may be omitted, replaced with one or more aspects of another example (e.g., Example 2), or combined with aspects of another example. The following is a non-limiting overview of some example implementations presented herein.

[0334] Example 1. A method comprising:

[0335] Generate one or more Generative Language Model (GLM) hints related to the test software program, wherein the one or more GLM hints are based on at least one or more hint templates, wherein the one or more hint templates are filled with at least program information describing the software program and requirement information describing the requirements of the software program;

[0336] Based at least on one or more outputs of the GLM corresponding to the one or more GLM prompts, generate a test architecture corresponding to testing whether the software program meets the requirements; and

[0337] The software program shall be tested at least based on the aforementioned test architecture.

[0338] Example 2. According to the method described in Example 1, wherein:

[0339] Generating one or more GLM prompts includes: generating a first prompt by at least populating a first prompt template based on at least the program information and the requirement information; and

[0340] The one or more outputs of the GLM include a first output, which is at least based on the first prompt and identifies one or more segments of program information corresponding to the requirement.

[0341] Example 3. According to the method described in Example 2, wherein:

[0342] Generating the one or more GLM prompts includes: generating a second prompt by at least filling a second prompt template based on the requirement and the one or more fragments; and

[0343] The one or more outputs of the GLM include a second output, which is at least based on the second prompt and identifies a test specification corresponding to the requirement and included in the test architecture.

[0344] Example 4. According to the method described in Example 3, wherein:

[0345] Generating the one or more GLM hints includes: generating a third hint by at least populating a third hint template based on at least the one or more fragments and the test specification; and

[0346] The one or more outputs of the GLM include a third output, which is at least based on the third hint and identifies test code corresponding to the test architecture.

[0347] Example 5. The method according to Example 4 further includes: storing one or more of the following as one or more entries in the repository:

[0348] The first output of the GLM is stored in association with the requirement information and the program information;

[0349] The second output of the GLM is stored in association with the demand information and the one or more fragments; or

[0350] The third output of the GLM is saved in association with the test specification.

[0351] Example 6. The method according to Example 1 further includes: evaluating the performance of the test on the software program.

[0352] Example 7. According to the method described in Example 6, wherein evaluating the performance of the test includes: determining the degree of code coverage corresponding to the test.

[0353] Example 8. The method described in Example 6 further includes: generating additional test code based at least on the evaluated performance.

[0354] Example 9. The method according to Example 7, wherein the additional test code is generated based on at least one or more additional GLM hints, which are based at least on the performance of the test.

[0355] Example 10. According to the method of Example 1, the program information includes one or more of the following:

[0356] Architecture documentation describing the components of the software program and the interactions between those components; or

[0357] Interface specification documents describing the specific implementation details corresponding to the components.

[0358] Example 11. A system comprising:

[0359] One or more processors, said one or more processors being configured to perform operations, said operations including:

[0360] The first prompt is generated based at least on the first prompt template, program information describing the software program, and requirement information describing the requirements of the software program;

[0361] Extract one or more pieces of program information related to the requirement based at least on the first output of a generative language model (GLM), wherein the first output of the GLM is based at least on the first prompt;

[0362] The second prompt is generated based at least on the second prompt template, the requirement information, and one or more fragments;

[0363] The test specification corresponding to the requirement is obtained based at least on the second output of the GLM, wherein the second output of the GLM is based at least on the second prompt;

[0364] The third prompt is generated based at least on the third prompt template, the one or more fragments, and the test specifications;

[0365] The test implementation method corresponding to whether the software program meets the requirements is obtained based at least on the third output of the GLM, wherein the third output of the GLM is based at least on the third prompt; and

[0366] The software program shall be tested at least based on the aforementioned test implementation method.

[0367] Example 12. The system according to Example 11 further includes storing one or more of the following as one or more entries in a repository:

[0368] The first output of the GLM is stored in association with the requirement information and the program information;

[0369] The second output of the GLM is stored in association with the demand information and the one or more fragments; or

[0370] The third output of the GLM is saved in association with the test specification.

[0371] Example 13. The system according to Example 11, wherein the program information includes one or more of the following:

[0372] Architecture documentation describing the components of the software program and the interactions between those components; or

[0373] Interface specification documents describing the specific implementation details corresponding to the components.

[0374] Example 14. The system according to Example 11, wherein the operation further includes: generating additional test code based at least on the performance of the test of the software program.

[0375] Example 15. The system according to Example 11, wherein the second template includes one or more example test specifications.

[0376] Example 16. The system according to Example 11, wherein the third template includes one or more example test implementations.

[0377] Example 17. The system according to Example 11, wherein the system is included in at least one of the following:

[0378] Control systems for autonomous or semi-autonomous machines;

[0379] Sensing systems for autonomous or semi-autonomous machines;

[0380] A system used to perform simulation operations;

[0381] Systems used to perform digital twin operations;

[0382] A system for performing optical transmission simulation;

[0383] A system for collaborative content creation of 3D assets;

[0384] A system used to perform deep learning operations;

[0385] A system for presenting at least one of augmented reality content, virtual reality content, or mixed reality content;

[0386] A system used to host one or more real-time streaming applications;

[0387] Systems implemented using edge devices;

[0388] Systems implemented using robots;

[0389] Systems used to perform conversational AI operations;

[0390] A system for performing one or more generative AI operations;

[0391] A system that implements one or more large language model LLMs;

[0392] A system that implements one or more Visual Language Models (VLMs);

[0393] A system that implements one or more multimodal language models;

[0394] A system for generating synthetic data;

[0395] A system containing one or more virtual machines (VMs);

[0396] A system that is at least partially implemented in a data center; or

[0397] A system that utilizes cloud computing resources at least in part.

[0398] Example 18. One or more processors, including:

[0399] Processing circuitry, the processing circuitry being used to perform operations, the operations including:

[0400] The first prompt is generated by filling a first prompt template with at least program information describing the software program and requirement information describing the requirements of the software program, wherein the first prompt corresponds to the portion of the program information that identifies the requirements.

[0401] Extract one or more fragments of the program information related to the requirement, based at least on the first output of the generative language model (GLM) corresponding to the first prompt;

[0402] The second prompt is generated by filling a second prompt template based at least on the requirement and the one or more fragments, the second prompt corresponding to identifying the behavior of the software program associated with satisfying the requirement;

[0403] The test specification corresponding to the requirement is obtained at least based on the second output of the GLM corresponding to the second prompt;

[0404] A third prompt is generated by filling a third prompt template based on at least one or more of the aforementioned fragments and the test specification, the third prompt corresponding to an implementation of the test specification;

[0405] The test implementation method corresponding to testing whether the software program meets the requirements is obtained based at least on the third output of the GLM corresponding to the third prompt; and the test is performed on the software program based at least on the test implementation method.

[0406] Example 19. One or more processors according to Example 18, wherein the operation further includes: generating additional test code based at least on the performance of the test of the software program.

[0407] Example 20. One or more processors according to Example 19, wherein the additional test code is generated based on at least one or more additional GLM hints, which are based at least on the performance of the test.

Claims

1. A method comprising: Generate one or more Generative Language Model (GLM) hints related to the test software program, wherein the one or more GLM hints are based on at least one or more hint templates, wherein the one or more hint templates are filled with at least program information describing the software program and requirement information describing the requirements of the software program; At least based on one or more outputs of the GLM corresponding to the one or more GLM prompts, generate a test architecture corresponding to testing how well the software program meets the requirements; as well as The software program shall be tested at least based on the aforementioned test architecture.

2. The method according to claim 1, wherein: Generating one or more GLM prompts includes: generating a first prompt by at least populating a first prompt template based on at least the program information and the requirement information; and The one or more outputs of the GLM include a first output, which is at least based on the first prompt and identifies one or more segments of program information corresponding to the requirement.

3. The method according to claim 2, wherein: Generating the one or more GLM prompts includes: generating a second prompt by at least filling a second prompt template based on the requirement and the one or more fragments; and The one or more outputs of the GLM include a second output, which is at least based on the second prompt and identifies a test specification corresponding to the requirement and included in the test architecture.

4. The method according to claim 3, wherein: Generating the one or more GLM hints includes: generating a third hint by at least populating a third hint template based on at least the one or more fragments and the test specification; and The one or more outputs of the GLM include a third output, which is at least based on the third hint and identifies test code corresponding to the test architecture.

5. The method according to claim 4, further comprising: Save one or more of the following as one or more entries in the repository: The first output of the GLM is stored in association with the requirement information and the program information; The second output of the GLM is stored in association with the demand information and the one or more fragments; or The third output of the GLM is saved in association with the test specification.

6. The method according to claim 1, further comprising: Evaluate the performance of the test performed on the software program.

7. The method of claim 6, wherein evaluating the performance of the test comprises: Determine the extent of code coverage corresponding to the test.

8. The method according to claim 6, further comprising: Additional test code should be generated based at least on the performance being evaluated.

9. The method of claim 7, wherein the additional test code is generated based on at least one or more additional GLM hints, the one or more additional GLM hints being based at least on the performance of the test.

10. The method of claim 1, wherein the program information includes one or more of the following: Architecture documentation describing the components of the software program and the interactions between those components; or Interface specification documents describing the specific implementation details corresponding to the components.

11. A system comprising: One or more processors, said one or more processors being configured to perform operations, said operations including: The first prompt is generated based at least on the first prompt template, program information describing the software program, and requirement information describing the requirements of the software program; Extract one or more pieces of program information related to the requirement based at least on the first output of a generative language model (GLM), wherein the first output of the GLM is based at least on the first prompt; The second prompt is generated based at least on the second prompt template, the requirement information, and one or more fragments; The test specification corresponding to the requirement is obtained based at least on the second output of the GLM, wherein the second output of the GLM is based at least on the second prompt; The third prompt is generated based at least on the third prompt template, the one or more fragments, and the test specifications; The test implementation method corresponding to the degree to which the software program meets the requirements is obtained, based at least on the third output of the GLM, wherein the third output of the GLM is based at least on the third prompt; and The software program shall be tested at least based on the aforementioned test implementation method.

12. The system of claim 11, further comprising storing one or more of the following as one or more entries in a repository: The first output of the GLM is stored in association with the requirement information and the program information; The second output of the GLM is stored in association with the demand information and the one or more fragments; or The third output of the GLM is saved in association with the test specification.

13. The system of claim 11, wherein the program information includes one or more of the following: Architecture documentation describing the components of the software program and the interactions between those components; or Interface specification documents describing the specific implementation details corresponding to the components.

14. The system of claim 11, wherein the operation further comprises: Additional test code is generated based at least on the performance of the tests performed on the software program.

15. The system of claim 11, wherein the second template comprises one or more example test specifications.

16. The system of claim 11, wherein the third template comprises one or more example test implementations.

17. The system of claim 11, wherein the system comprises at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system used to perform deep learning operations; A system for presenting at least one of augmented reality content, virtual reality content, or mixed reality content; A system used to host one or more real-time streaming applications; Systems implemented using edge devices; Systems implemented using robots; A system for performing conversational AI operations; A system for performing one or more generative AI operations; A system that implements one or more large language model LLMs; A system that implements one or more Visual Language Models (VLMs); A system that implements one or more multimodal language models; A system for generating synthetic data; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

18. One or more processors, comprising: Processing circuitry, the processing circuitry being used to perform operations, the operations including: The first prompt is generated by filling a first prompt template with at least program information describing the software program and requirement information describing the requirements of the software program, wherein the first prompt corresponds to the portion of the program information that identifies the requirements. Extract one or more fragments of the program information related to the requirement, based at least on the first output of the generative language model (GLM) corresponding to the first prompt; The second prompt is generated by filling a second prompt template based at least on the requirement and the one or more fragments, the second prompt corresponding to identifying the behavior of the software program associated with satisfying the requirement; The test specification corresponding to the requirement is obtained at least based on the second output of the GLM corresponding to the second prompt; A third prompt is generated by filling a third prompt template based on at least one or more of the aforementioned fragments and the test specification, the third prompt corresponding to an implementation of the test specification; At least based on the third output of the GLM corresponding to the third prompt, obtain a test implementation method corresponding to the test program's satisfaction of the requirement; and The software program shall be tested at least based on the aforementioned test implementation method.

19. The processor of claim 18, wherein the operation further comprises: Additional test code is generated based at least on the performance of the tests performed on the software program.

20. One or more processors according to claim 19, wherein the additional test code is generated based on at least one or more additional GLM hints, the one or more additional GLM hints being based at least on the performance of the test.

Images