Automatic Differentiation and Optimization of Heterogeneous Simulation Intelligence Systems

Abstract

Provided herein are systems and methods that include one or more virtual machines and a compiler for enabling automatic differentiation across a stack. Further provided herein are techniques, integrations, and workflows that enable automatic differentiation.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 327,145, filed April 4, 2022, which is incorporated by reference in its entirety. [Background technology]

[0002] The main classes of applications driving extreme-scale parallel and distributed computing, as well as new hardware architectures for domain or application-specific accelerated computing, e.g., machine learning (ML) and artificial intelligence (AI), data engineering and data analytics, and simulation, are highly interdependent, especially in the context of data-driven learning systems and scientific and engineering domains. Thus, there is a need for the development of systems and methods for the automation of software and / or hardware development, including automated or dynamic configuration and optimization of heterogeneous components across the hardware-software stack. In addition to the exploding complexity of hardware-software stack configurations for a given AI / ML or simulation application, there are information bottlenecks (e.g., vertical boundaries) between stack abstraction levels that hinder cross-layer interactions, let alone optimization. These concerns must be addressed in new integrated system stacks purpose-built for AI / ML and simulation workloads. Summary of the Invention

[0003] This disclosure involves the design and development of a new class of runtime environments built from first principles for (and with) Machine Learning (ML) and Artificial Intelligence (AI). The goal of this runtime environment includes integrating the latest AI / ML techniques into the operating system itself to achieve substantially better performance for a task. This is possible if such a system can dynamically tune the entire software and / or hardware stack so that it can solve problems faster when presented with them. Optimizations are made to perform computations to solve the task more efficiently and reliably.

[0004] Examples of such behaviors that dynamically adjust the stack include, but are not limited to, dynamic just-in-time (JIT) program generation, compilation, and evaluation (e.g., program synthesis, solution-space exploration); differentiable components of a runtime system that enable dynamic resource allocation (e.g., memory management, processor allocation, etc.) so that tasks are completed more quickly or in a more efficient manner; or operating system integration with user applications via a standard API (e.g., the user does not modify anything compared to other systems) and additional APIs (e.g., the user can inform the OS of their intentions).

[0005] Additionally, the runtime environment of the present disclosure may enable AI algorithms that are currently infeasible. Examples of why some AI algorithms may be infeasible include, but are not limited to, computational inefficiencies at too large a scale, low-level primitives that do not allow for probabilistic inference operations, or because valuable information is lost at the abstraction layers of the software-hardware stack.

[0006] Methods to overcome such challenges include building a novel AI-native OS, where information can be propagated vertically in the software-hardware (SW-HW) stack for AI / ML. Thus, disclosed herein is a novel system of autodifferentiation (autodiff or AD) up and down the entire stack to compute, expose, and propagate gradients, errors, etc. Several overarching themes arise when considering such a system that involves autodifferentiation up and down the entire stack. Such themes include, but are not limited to, optimization of any task, gradient-based learning across SW-HW abstraction levels, ML and probabilistic reasoning in close-to-the-metal (via probabilistic programming, differentiable programming, and autoconfiguration DSLs), new dimensions of uncertainty reasoning, reliability of new metrics, etc., and optimization of heterogeneous hardware.

[0007] In one aspect, disclosed herein is a system comprising one or more virtual machines (VMs) configured to generate a reconfigurable architecture, each of the one or more VMs comprising an abstraction of a computer engine, and a compiler configured to compile software to be executed on the one or more VMs, the compiler configured or capable of autodiffing. In some embodiments, the one or more VMs comprise an autodiff function. In some embodiments, the autodiff comprises emitting a gradient program in an intermediate representation (IR) and / or an instruction set. In some embodiments, the one or more VMs are parameterized. In some embodiments, the one or more VMs comprise one or more parameters to be optimized. In some embodiments, the one or more parameters comprise a size of memory, a number and / or width of registers, an available instruction set, an instruction encoding, a firmware implementation, input / output (I / O), or any combination thereof. In some embodiments, each of the one or more VMs comprises private resources. In some embodiments, the compiler includes a multi-level intermediate representation (MLIR) compiler or a low-level intermediate representation (LLVM IR) compiler. In some embodiments, the automatic differentiation includes translating a programming language into machine code that is processed on heterogeneous hardware. In some embodiments, the compiler enables the use of probabilistic programming, domain-specific languages, differentiable programming, or any combination thereof. In some embodiments, one or more VMs run on a GPU, a CPU, an FPGA, an ASIC, an NVMe, a microcontroller, an AI accelerator, or any combination thereof. In some embodiments, the AI ​​accelerator includes a Google-TPU®, Graphcore®, Cerebras®, SambaNova®, or a combination thereof. In some embodiments, about 1000 VMs run on a GPU.In some embodiments, about 10,000 VMs run on the GPU. In some embodiments, about 500 VMs run on the CPU. In some embodiments, each of the one or more VMs is about 100 to about 500 lines of code. In some embodiments, each of the one or more VMs is about 300 lines of code. In some embodiments, the system enables machine programming across the stack. In some embodiments, the system further comprises a software stack, a hardware stack, or a hardware-software stack. In some embodiments, the software stack, the hardware stack, the hardware-software stack, or any combination thereof is differentiable. In some embodiments, the software stack, the hardware stack, the hardware-software stack, or any combination thereof enables gradient-based learning and / or optimization. [Brief description of the drawings]

[0008] A better understanding of the features and advantages of the present subject matter will be obtained by reference to the following detailed description that sets forth illustrative embodiments and the accompanying drawings. [Figure 1] 1 shows a non-limiting example of a block diagram of a system with a Sim-AI OS. [Diagram 2] 1 shows a non-limiting example of a block diagram of the Sim-AI OS. [Diagram 3] 1 illustrates a non-limiting example of a branch space for machine programming. [Figure 4] A non-limiting example of a computer hardware and software configuration is shown. [Diagram 5] FIG. 1 illustrates a non-limiting example of a computer architecture. [Figure 6] 1 illustrates a non-limiting example of a block diagram of a virtual machine having a flexible network topology. [Figure 7]1 illustrates a non-limiting example of flow of gradients in a block diagram of a virtual machine with a flexible network topology. [Figure 8] 1 illustrates a non-limiting example block diagram of a low-level intermediate representation used by a compiler framework. [Figure 9] 1 shows a non-limiting example of a block diagram with memory contents of a grid of 16 VMs executing an 18-byte long program that generates a selected pattern. [Figure 10] 1 shows a non-limiting example of an FPGA floorplan illustrating VMs placed in programmable logic. [Figure 11] 1 diagrammatically illustrates a computer system that is programmed or otherwise configured to implement the methods provided herein. Invention Details

[0009] The disclosed technology includes an ideal area for development and evaluation (and future dissemination and education) at the intersection of accelerated (and high performance) computing, AI, and engineering sciences. The technology includes use cases in many important areas of science and engineering, providing useful but challenging constraints and variability for the technology described herein (and machine programming and AI / ML, broadly). As an example, heterogeneous settings are the norm (both in heterogeneous domain types and system types). And in simulation intelligence (SI) environments, where differences such as scientific data and hypothesis-based workflows require clear and novel definitions of performance and reliability, common ML evaluations and metrics are inadequate. In addition, by pursuing HW-SW innovation in this way, there may be a flywheel effect, because improvements in SI methods with well-constructed use cases may lead to novel computational architectures (and possibly substrates), which in turn improve the fundamental SI methods and the platforms on which they are deployed. In particular, the present disclosure includes full-stack autodiff, HW-SW differentiable programming, and AI-native OS (e.g., similar to FIG. 1).

[0010] Like the impact that automatic differentiation (and more specifically, differentiable programming) has had on the machine learning and scientific computing communities, the introduction of such capabilities into the broader computational stack is expected to have a similar transformational or paradigm shift. For example, deep learning, the renaissance of machine learning this century, may not be possible without automatic differentiation and differentiable programming. Thus, the advances and techniques described herein may have a transformative effect on the scientific and engineering process.

[0011] The benefits of MP, full hardware-software stack gradient-based learning, and (among others) OS-level autodiff can not only significantly accelerate existing computational workflows, but also allow practitioners to consider problems and challenges from completely new perspectives. With MP and DP as core elements ("motifs") of the new Simulation Intelligence (SI) paradigm, MP and DP can be expected to play a key role in shaping the SI stack (e.g., Figure 1) and the subsequent reshaping of the scientific method leading up to the Nobel-Turing breakthroughs.

[0012] In some embodiments, the techniques of the present disclosure can be used with the Simulation-AI Operation System (Sim-AI OS) and its accompanying workflows and data flows. In some cases, the Sim-AI OS can be used interchangeably with the Simulation Intelligence Operating System (SIOS). In some cases, the Sim-AI OS and its accompanying workflows and data flows can be dedicated to simulation intelligence, general purpose for use across scientific and intelligence domains, or both. In some cases, the systems of the present disclosure enable optimal, efficient, and novel workflows for several user groups in fields including, but not limited to, engineering, science, and intelligence.

[0013] Machine Programming (MP) The disclosed system comprises machine programming (MP) for dynamically and efficiently configuring a hardware-software stack of elements that may be highly heterogeneous, combining a myriad of types of hardware accelerators, computing architectures (up to petascale and exascale), various programming languages, and mixed data types. As an example, MP comprises an essential component (or rather a "motif") of the new field of Simulation Intelligence (SI) (e.g., the integration of AI / ML, simulation, and scientific computing in new synergistic ways). The main classes of applications that drive extreme-scale parallel and distributed computing machine learning, data analysis, and simulation are, unlike SI, highly interdependent and in a complex manner across essentially all domains. In one aspect, a key feature of SI that makes the system described herein relevant to scalable computing and engineering science is that its manifestation suggests a rethinking of the software-hardware stack (Figure 1). In another aspect, a key feature of SI that makes the system described herein relevant to scalable computing and engineering science includes its applicability across scientific, engineering, and social disciplines.

[0014] Real-world applications applicable to the disclosed system may include several factors. In some cases, they may be similar enough to utilize the same research methods as constructed herein. In some instances, they may be differentiated in such a special way that they force generalization of the research methods (e.g., different applications impose separate requirements on deployment settings, hardware heterogeneity, data types and sizes, time and space efficiency requirements, etc.). In some instances, the scientific results may be important to a given field apart from technological advances such as MP. As an example, it may be possible to accelerate a coastal climate simulator 1000x to enable real-time (<10 minutes) disaster preparation, allowing scientists to iterate more quickly across experimental scales to ask questions that were not possible before). In some instances, SI may be leveraged to make further interdisciplinary advances. As an example, they may integrate multiple scaling operations of computational science workflows (e.g., materials design, synthesis, validation, production), as well as the ability to learn and self-improve, because gradient-based algorithms can perform various optimizations that are currently unthinkable because existing workflows do not propagate information and would stall out completely.

[0015] MP generally involves a new paradigm in computing research, towards the automation of software and / or hardware development. From the perspective of simulation in science and engineering, systems will soon require, if not already, the automation aspect of software development, especially when considering data-driven learning systems. MP is therefore a motif of Simulation Intelligence (SI). With MP in the SI operating system (Figure 1) as a motif interfacing hardware and software, the integration of SI motifs can become much more efficient (e.g., automatic HW-SW optimization and codesign), robust (e.g., less prone to human and stochastic errors), and scalable (e.g., modular, flexibly deployable from HPC to the edge).

[0016] As shown in FIG. 1, MP can understand code across motifs, compose them into working systems, and then optimize these systems to run on the underlying hardware. Thus, while MP may not be tied to any particular class of SI problem, MP is essential as the glue that ties other SI motifs together and acts as a bridge to the hardware. MP, along with the novel vertical integration and differentiability described herein, can further propagate valuable uncertainties, as well as error and / or reliability metrics between the hardware and software layers. Furthermore, information surfaced from such MP integration can surface valuable hardware-software insights. As an example, valuable hardware-software insights can include providing full-stack gradient data that can be used for reverse design of new computing architectures and boards.

[0017] Without MP operating in a cohesive and / or holistic full-stack fashion, bottom-up constraints may arise on the types of data structures and algorithms implemented on top of the hardware layer. As an example, this is not unlike "hardware lottery," where research ideas and techniques succeed and dominate because they are suited to the available software and hardware, not necessarily because they are better than other directions. This "hardware lottery" is thus crucial in the paths pursued by the largely disconnected hardware, systems, and algorithm communities. As a further example, the past decade has provided industry-shaping examples of deep learning and GPUs. However, the various mechanisms of MP may provide flexible use of heterogeneous software components, as well as abundant opportunities for mitigating such hardware lottery effects in SI, and in AI and software more broadly.

[0018] Provided herein are systems and methods for delivering results in SI applications that each (a) integrate multiple AI / ML methods, (b) enable simulation with multiple types of real-world data, (c) enable scientific computing in multidisciplinary environments, and (d) leverage more challenging and / or scientifically promising components of the hardware-software stack, such as close-to-the-metal domain-specific languages, multi-language compilers, heterogeneous hardware, etc., or any combination thereof.

[0019] Open benchmarks of MPs may be needed to collectively and reproducibly address challenges related to the various mechanisms of MPs. In some instances, this is similar to areas of the ML community such as computer vision and natural language processing. However, in some cases, generating quality benchmark MPs may be more challenging. Such reasons include, but are not limited to, (a) because the exact differences between computing components in one environment and another directly affect the evaluation results (e.g., a 4-GPU instance of Amazon® in Virginia is equivalent to a 4-GPU instance of Microsoft® in London, although this is less relevant in computer vision or NLP), (b) because problem-specific evaluation methods are often useful (e.g., rather than a one-size-fits-all dataset), and (c) because methods are not necessarily comparable (e.g., Figure 3), as many of them fall in the spectrum of ML-to-formal methods.

[0020] In some cases, MP benchmarks are provided herein that establish a principled, efficient, understandable testing ground that can help tune and advance the field. As an example, this is similar to the "ImageNet moment" of computer vision from a decade ago. However, in some cases, care is taken to avoid inherent limitations, errors, and biases (such as those present in ImageNet). As a further example, it is undesirable to allow a particular dataset or narrow problem within a field to shape how the field is defined as a result.

[0021] MP developments and applications provided herein aim to demonstrate an order of magnitude increase in scientific programming productivity. In some instances, this is accomplished by making more / all parts of the software lifecycle more efficient, including but not limited to reducing the time spent tracking down software quality defects such as those related to correctness, performance, safety, and portability. In some instances, MP can enable scientists, engineers, technicians, and students to produce and maintain high quality software as part of their problem-solving process without requiring specialized software development skills, which today can be a choke point in researcher and engineering organizations in many disconnected capacities.

[0022] MP enables techniques that can understand code across programming languages ​​and / or model types. MP can further automatically synthesize programming languages ​​and / or model types into working systems and optimize these systems to run on underlying hardware. In some examples, the underlying hardware is homogenous. In some examples, the underlying hardware is heterogeneous. While some forms of intentional communication from humans to machines are emerging (e.g., translation from natural language to programming language, and translation of visual diagrams into software programs such as GitHub's Co-Pilot®, or automated synthesis of code for visual diagrams), the scope of MP capabilities can be realized with full-stack differentiablity of software hardware. In some cases, this is especially true for general purpose use and deployment with heterogeneous systems.

[0023] The current state of the MP field applies machine learning techniques (e.g., deep learning) to learn high-dimensional mappings between code inputs and outputs. In some examples, this includes learning models to automate software debugging (e.g., in the control space comprehensively covered by a given dataset). In such examples, this automation is achieved in part by gradient-based learning on software programming datasets. In comparison, the full-stack automatic differentiation provided herein can enable another dimension of MP. This includes making the stack itself differentiable. In some examples, making the stack differentiable enables gradient-based learning at multiple levels of abstraction across the full software-hardware stack (e.g., rather than isolated to one particular layer or component of the stack). In some examples, making the stack differentiable then enables learning models for manual, suboptimal, and error-prone resource allocation and other stack operations (e.g., referred to as MP2). In some examples, even proof-of-concept demonstrations of MP2 have important implications for computing systems broadly and for high-efficiency SI stacks (Figure 1). In some examples, this is true in AI / ML as well.

[0024] Simulation AI (Sim-AI) Operating System (OS) The machine programming and / or autodiff functionality of the present disclosure can exist within a software hardware stack framework that includes an operating system (OS) and surrounding technology specialized for simulation intelligence (e.g., FIG. 1). In some examples, the OS and surrounding technology are also general purpose for scientific and intelligence workflows across domains. Referring to FIG. 1, an example block diagram of a system including a Simulation AI (Sim-AI) Operating System (OS) is shown. The system may include a Simulation Intelligent (SI) stack, which may enable optimal, efficient, and novel workflows for several user groups (105) in engineering, science, and intelligence. Users (105) may generally include scientists and other researchers across many domains and scales (e.g., from particle physics and biology to materials design and drug discovery, weather and climate science, socio-economics and geopolitics, astrophysics and cosmology), such as ML / AI scientists and engineers, data scientists, theorists, roboticists, AI scholars, automation scientists (or “self-driving laboratories”), information analysts, economists, and political scientists, or other domain experts including, but not limited to, policy and decision makers (from climate to economics to financial / trading markets). In some embodiments, as shown in FIG. 1, the SI stack may include modules and interfaces for users (105) to interact with at various levels of abstraction, such as top-level workflows (110), low-level machine programming interfaces and modules (125), and programmatic and user-friendly interfaces for flexible and configurable ML and data modules (115).

[0025] In some embodiments, the SI stack may comprise a computing and data infrastructure having one or more layers, including a workflow layer (110) (e.g., applications), a simulation artificial intelligence (Sim-AI) operating system (OS) (115) (e.g., simulation module layer (120)), a machine / intentional programming (MIP) layer (125), and a hardware layer (130) (e.g., computational boards such as CPUs, GPUs, FPGAs, etc.). In some cases, the Sim-AI OS (115) may comprise one or more engines (e.g., techniques and / or methods). In some cases, the MIP layer (125) may be any programming interface (e.g., manual or machine programming).

[0026] The Sim-AI OS (115) may comprise one or more classes of complementary algorithmic methods that represent foundational techniques for synergistic simulation and AI technologies. In some cases, the workflow layer (110) may comprise one or more Sim-AI workflows (e.g., applications) that may be the result of integration with the Sim-AI OS (115). In some examples, the workflow layer (110) may exchange information, feedback, and data that informs the development and implementation of the Sim-AI OS (115). In some examples, the workflow layer (110) may comprise new scientific and intelligence workflows, such as, but not limited to, automation and human-machine teaming processes (e.g., "active sciencing," inverse problem solving, etc.). The Sim-AI OS (115) and SI stack manifestation of FIG. 1 comprises a modular and versatile software package that, as is, can provide structured layers of the workflow described herein while also providing users with the flexibility and extensibility to build on and / or substitute custom components.

[0027] Moreover, the information flows illustrated in FIG. 1 may not be linear or mutually exclusive. There may be any number of variations of information flows in the SI stack of FIG. 1. That is, FIG. 1 may represent a general scheme in which there may be numerous variations that may combine methods for similar information flows and workflows. As an example, an ML researcher may implement a pipeline that traverses multiple layers (and information paths). As a further example, if a user who is an engineer is working with machine programming, the engineer may work with Sim-AI OS methods simultaneously or sequentially.

[0028] The SI stack of Figure 1 may also provide interfaces and full pluggability for third-party, proprietary, open source, open standard, custom, or other types of extensions. Extensions may include, by way of non-limiting examples, additional machine learning libraries, hardware-software codesigns, software development kits (SDKs), application programmable interfaces (APIs), computation distribution, privacy-preserving algorithms, and data pipelines.

[0029] A set of technology capabilities enabled by the present disclosure that can help scientific endeavors and fields overcome bottlenecks related to simulation, intelligence, and data include, by way of non-limiting examples: ● Synergistic combination of expert knowledge models (first principles, dynamics equations, etc.) and data-driven surrogate models - Flexible combination of multiple engines / simulators An efficient and scalable simulator that operates at multiple fidelity and abstraction levels Simulate and learn processes and systems across multiple physics and multiple spatial and temporal scales Possible / reliable simulations and inferences with limited data - Verify the simulation with real data and quantify reliability (uncertainty, etc.) Learning a model that reflects a probabilistic causal structure Easily incorporate domain expertise and causal structures into the simulator Generate counterfactual simulations (e.g., based on causal theory) that reliably simulate possible futures for large timescales and high-dimensional, multi-modal data. ●Solving inverse problems in other difficult spaces and domains Accelerate existing simulators many times over without refactoring or rewriting legacy codebases Spinning up a simulator with direct physical analogy Providing a catalog of domain-specific engines and workflows (e.g., for non-ML experts) Seamless integration with real-world datasets and online data streams Data engineering to ensure reliability, reproducibility, and safety ● Systems AI: Systems engineering and uncertainty reasoning for complex and dynamic combinations of data, software, hardware, and people. may include.

[0030] The Sim-AI OS (115) may comprise a simulation module layer (120) that may enable one or more workflows described herein. The simulation module layer (120) may utilize accelerated computing building blocks in the hardware layer (130). In some embodiments, the simulation module layer may comprise causal computation (e.g., causal ML), agent-based modeling, open-endedness, working memory, domain-specific languages ​​(DSLs) (e.g., SI DSL for probabilistic simulation), semi-mechanical modeling, surrogate modeling and emulation, multiphysics modeling, multi-scale modeling (e.g., spatial and temporal), and simulation-based inference. In some cases, the Sim-AI module may be combined with other simulation methods and applications (e.g., computational fluid dynamics (CFD), finite element analysis (FEA) solvers, etc.). In further cases, the Sim-AI module may be integrated with existing machine learning frameworks and tools (e.g., PyTorch® and TensorFlow® libraries for tensor modeling and autodiff).

[0031] 2, modules in the simulation module layer (220) of the Sim-AI OS (205) may be built using an engine (215) (e.g., techniques and / or methods). In some embodiments, the engine (215) may comprise a probabilistic programming (220) framework, a differentiable programming (225) framework, or a combination thereof. In some embodiments, the two frameworks referred to herein may include overlaps or interconnections (e.g., a probabilistic programming language may use an automatic differentiation mechanism in differentiable programming).

[0032] In some embodiments, the engine (215) may comprise a probabilistic programming (220) framework. Probabilistic programming may be used to extend the scope of probabilistic graphical models (PGMs) by enabling inference across arbitrary models as programs (e.g., programming constructs such as recursion and stochastic branching). Probabilistic programming may allow generative models to be easily written and experimented with. In some cases, the probabilistic programming (PP) paradigm may equate probabilistic generative models with executable programs. Probabilistic programming languages ​​(PPLs) allow experts to leverage the power of a programming language to generate rich and complex models, while relying on a built-in inference backend that operates on any model written in the language. This decoupling may be a powerful abstraction, as it may enable experts to rapidly iterate across models and experiment in prototyping through deployment. Programs may be generative models, as they may relate unobservable causes to observable data to simulate how data is created in the real world.

[0033] Similarly, differentiable programming is a generalization of neural networks and deep learning that allows gradient-based optimization of arbitrarily parameterized program modules.

[0034] These can provide a much more expressive toolkit for knowledge representation.

[0035] As an example, in a probabilistic program, one can define a joint distribution p(x,y) of latent (unobserved) variables x and observed variables y. A PPL inference engine can generate the posterior distribution p(x|y)=p(y|x)p(x) / p(y) of the unobserved variables given the observed variables or data. Because the observed variables can correspond to the outputs of a program, probabilistic programming can provide a way to "invert" a given program, meaning that the inputs of the program can be inferred given instances of data that correspond to the program's outputs. This can be contrasted with standard programs, which can take inputs to produce the corresponding outputs.

[0036] In some embodiments, a probabilistic program may be used as a simulator. In some cases, a probabilistic program may itself be a simulator since it can represent a probabilistic generative model of data. In such cases, the simulated data may be generated by forward execution of the program, given a system or process well described in the PPL. In some examples, the simulator may be forwarded to predict the future, but the probabilistic program parameters may be inferred based on observed outcomes. As an example, a neurodegeneration program may use random sampling to define a statistical model from which probabilistic disease trajectories may be generated and parameters may be inferred for individual biomarkers. Furthermore, simulating with a probabilistic program may enable semi-mechanistic modeling, and a Turing-complete PPL may provide flexible modeling to integrate the physical laws or mechanistic equations describing the system with data-driven components that may be tuned or trained against data observations. In some examples, a PPL may enable semi-parametric modeling, and an inference engine may handle programs that combine parametric and non-parametric models. In such instances, the parametric component can learn a fixed number of parameters and allow the user to specify domain knowledge, while the non-parametric model (e.g., Gaussian Process) can learn an infinite number of parameters that grow with training data. These two programming characteristics, semi-mechanistic and semi-parametric modeling, along with the high overall expressiveness of PPL, may be well suited for non-ML experts to easily build and experiment with probabilistic inference algorithms.

[0037] In some examples, PPL and DP may be used for "partially specified" models with black boxes (e.g., deep learning) and fully specified simulators. In further examples, PPL may be advanced and scaled in simulation-based inference. This may be especially true in the context of human-machine inference. Using probabilistic programs as simulators may provide advantages such as, but not limited to, modularity in the modeling system, general-purpose inference engines, uncertainty quantification and propagation, and interpretability.

[0038] In some embodiments, the engine (215) may comprise a differentiable programming (225) framework. Differentiable programming (DP) may comprise a programming paradigm in which derivatives of a program are automatically calculated and may be used in gradient-based optimization to adjust the program to achieve a given objective. DP may be used in a wide variety of fields, such as scientific computing and ML. In some embodiments, DP may be considered a generalization of deep learning. In some cases, the potential of DP may be enormous when taken beyond deep learning to the general case of complex programs. In some instances, existing programs may take advantage of the vast amount of knowledge embedded within them. In some instances, this may enable the creation of novel models and the performance of in silico experiments.

[0039] In some examples, DP coupled with ML and simulation can be used to derive techniques for working with non-differentiable functions such as ReLU used in deep learning, or with non-smooth functions. In some examples, it may also be useful to work with approximate reals rather than reals and seek numerical accuracy theorems. In further examples, especially in ML, it may be important for DP to add explicit tensor (multidimensional array) kinds with accompanying shape analysis. Furthermore, richer DP languages ​​may support a wide range of types and computations (e.g., recursion, or programming with Riemannian manifolds (to accommodate natural gradient descent)). In some examples, Julia, C++ (e.g., DiffTaichi®), and close-to-the-metal domain-specific languages ​​can be used to make computation runtimes more efficient in scale.

[0040] With reference to FIG. 1, the Sim-AI OS layer (115) can be communicatively coupled to the workflow layer (110). In some embodiments, the modules and engines of the Sim-AI OS layer (115) can enable one or more workflows (or applications) of the workflow layer (115). In some embodiments, the workflow layer (115) is enabled by one or more modules of the Sim-AI OS. In some embodiments, the workflow layer may include inverse design (also called inverse problem solving), open-ended optimization, continuous learning, causal inference and discovery, simulator inversion, human-machine reasoning (also called active science), uncertainty reasoning, counterfactual reasoning, digital twins (e.g., with live data), multi-modal simulation, and / or physics-based learning. In some examples, the workflow may include online and dynamic optimization.

[0041] In some embodiments, a pathway between users (105), such as scientists and domain experts, and the workflow layer (110) may be beneficial for practical use. For example, for AI and ML scientists and engineers, this can be an important pathway for developing and utilizing the Sim-AI OS (115) and workflow layer (110). Users (105) may further include many areas of expertise and interest, and users may interact with any layer and component (e.g., a software infrastructure engineer working directly with HPC at the lower hardware level, an ML scientist working directly with probabilistic programming or open-ended without workflow context, etc.).

[0042] In this disclosure, Sim-AI OS modules may include significant and valuable overlaps with each other. Such overlaps provide a valuable framework for integrating such modules. Non-limiting examples of module integration may include the use of multi-scale and agent-based modeling (e.g., systems biology, sociopolitics, etc.), the use of physics-infused ML to build surrogate models in multi-physics scenarios, the use of simulation-based causal exploration in agent-based network models, or semi-mechanistic methods implemented for multiphysics modeling. In some embodiments, one or more elements of the simulation module layer (210) may be combined with one or more elements of the engine (215). In some embodiments, the SI stack of the present disclosure includes a dedicated framework for integrating the motifs (e.g., modules or engines) described herein. Table 1 provides a non-exhaustive view of the domains for use-inspired research for each motif, along with various methods and use cases resulting from the integration of the motifs.

[0043] [Table 1]

[0044] With reference to FIG. 1, implementations of Sim-AI OS modules and engine components (115), such as those described herein, can leverage available computing resources while also informing the design of new computing architectures. In some embodiments, the Sim-AI OS (115) as well as the MIP layer (125) can be communicatively coupled to a hardware layer (130) to enable SI as described herein. In some cases, the SI stack can comprise one or more hardware layers. In some embodiments, the Sim-AI pipeline described herein can be implemented at scale on high performance computers (HPC) and supercomputers, on simple 4-CPU laptops, anything in between, or combinations thereof. In some embodiments, the Sim-AI pipeline can be implemented on a centralized network, a decentralized network, a distributed network, or combinations thereof. In some cases, the network can comprise a cloud including one or more servers.

[0045] In some embodiments, the hardware interface layer (e.g., machine / intentional programming (125) and DSL) can add useful abstraction between low-level computer hardware and other layers, modules, and users. In such embodiments, information can flow between layers, modules, and users to generate programs and / or computationally optimized code that runs on the hardware (130) described below. In some embodiments, users (105) (e.g., scientists, engineers, analysts, domain experts, etc.) can easily develop and interact with low-level (or close-to-the-metal) code via the MIP (125) and DSL. In some cases, output from the machine programming can also communicate information (e.g., logs, feedback, etc.) to users (105).

[0046] In some cases, user software, system software, and the underlying computational substrate may be integrated through the MIP layer (125). In some examples, the SI stack may further include a hardware abstraction layer. In some examples, the hardware abstraction layer may perform just-in-time (JIT) compilation and integrate MP methods into the compiler as well as the runtime. In such cases, the Sim-AI OS (115) may provide OS-level performance estimation, dynamic AI-driven CPU / task allocation, data-driven scheduling, memory management, and cache policies and networking.

[0047] The machine programming or machine intentional programming (MIP) (125) layer described herein may enable extending artificial intelligence. Machine programming (MP) may generally comprise software (and hardware) automation. The MIP systems described herein may comprise systems that can automate some or all of the actions that turn a user's intent into an executable program and maintain that program over time. The present MIP implementations may be uniquely suited to the Sim-AI OS and scientific workflows. These may differ from mainstream ML workflows, such as by significantly affecting the flow of data, results, and end-user behavior. Furthermore, in addition to the Sim-AI OS accelerating science and intelligence workflows, MIP may further reduce development time through some degree of automation to reduce the cost of producing safe, accurate, correct, and efficient software. These systems may also enable non-programmers to harness the full power of modern computing platforms to solve complex problems accurately and efficiently.

[0048] In some embodiments, MP can reason across three distinct pillars: (i) intent, (ii) invention, and (iii) adaptation. Intent can include identifying new ways and simplifying existing ways for users to express their ideas to machines. It can also include extracting meaning (such as semantics) from existing software. The invention can include higher-level algorithms and data structures that can satisfy the user's intent. In some cases, the inventive system can simply fuse existing components together to form a novel solution that meets the user's intent. Adaptation can include taking an invented higher-level program and adapting it appropriately to a particular hardware and software ecosystem. This can be done to ensure that certain quality characteristics are maintained, such as correctness, performance, safety, and / or maintainability.

[0049] In some embodiments, the hardware layer (130) may be communicatively coupled to the MIP layer (125). In some cases, the substrate for computation may comprise commodity hardware (e.g., laptop), using pre-built libraries (e.g., BLAS), providing one-way interaction upwards. In alternative cases, the substrate for computation may comprise HPC grade (e.g., 1 PF / s CPU, GPU, etc.), which may be bidirectional interaction. In some examples, standard HPC software (e.g., for quantum computing) may be used with additional injection of machine programming (MP) to develop custom new software. In further cases, the substrate for computation may comprise specialized hardware (e.g., FPGA, ASIC, or other dynamically reconfigurable hardware, etc.), which may also provide bidirectional interaction. For example, one may build a simulation model of a CPU to create the model, select existing computation blocks (functions, instructions), and define new functions via MP. In some embodiments, the computation may be Will advances in motifs lead to new kinds of algorithms that require fundamentally new systems and architectures? Perhaps these could be implemented as ASICs (Application Specific Integrated Circuits), or would new substrates be useful? Will reverse design of SI yield novel computing architectures and substrates? • How will the open-ended paradigm affect computing systems (memory allocation, load balancing, caching, etc.)? ●How will machine programming impact hardware-software co-design for simulation intelligence (and high-performance computing)? In addition to goals of performance and scalability (e.g., latency, throughput, etc.), energy efficiency, and reliability, how do advancing technologies build abstractions to facilitate reasoning about correctness, safety, and privacy? • Will SI-based surrogate modeling enable a landscape shift in which computationally lightweight surrogates running on a few GPUs replace large supercomputing workloads (especially in physical and life science simulations)? What are the metrics and benchmarks for quantifying the tradeoffs between such orthogonal computation approaches? Considerations may be required, including, but not limited to:

[0050] Additionally, some embodiments may benchmark idealized neural network models and datasets using distributed training algorithms on HPC platforms. However, in some cases, benchmarking may not provide insight into the actual performance of these approaches in real deployment scenarios. Such cases may include, but are not limited to, the use of domain-inspired AI architectures and optimization schemes for data-driven discovery in the context of realistic datasets that may be noisy, incomplete, and heterogeneous.

[0051] Additionally, the simulation-AI methods, models, and / or physical calculations involved in the simulations described herein may incorporate optimization of sensor and chip designs, as well as training of NNs with physical components. In some embodiments, ML-enabled intelligent sensor design may be used with inverse design and related machine learning techniques to optimally design data collection hardware with respect to user-defined cost functions or design constraints. In some cases, such a process may include: 1. Standard engineering practice involves an initial design α o , or generate a randomized initialization α of parameters and spatial configuration (e.g., with respect to the transformation elements or sensing components). 2. Generate raw sensing data X from the initial α. i , y i and machine learning

[0052]

number

[0053]

number

[0054]

number

[0055] In some embodiments, such approaches or specific operations in the above-mentioned process can improve the overall performance of the sensor and the broader data capture / modeling system. In some cases, non-intuitive design choices can improve overall performance. As an example, in the field of synthetic biology, a surrogate-based design framework with neural networks trained on RNA sequence-function mapping can be designed to intelligently design RNA sequences that synthesize and perform specific molecular sensing tasks. This can result in an in silico design framework for engineering RNA molecules as programmable response elements to target proteins and small molecules. In such examples, data-driven approaches can outperform the predictive accuracy resulting from standard thermodynamic and kinetic modeling. In some cases, data-driven ML-based sensor design can outperform "intuitive" designs that may be based solely on analytical / theoretical modeling. In some examples, there may be non-trivial limitations, such as large, well-characterized training datasets that can be useful for engineering and selecting sensing features that can statistically separate various inherent noise terms or artifacts from the target signal of interest. In some further examples, there are limitations due to the curse of dimensionality, where the high dimensional space of the training data can drown out significant correlations with the target sensing information.

[0056] Automating and optimizing various processes may also include designing, manufacturing, and verifying computer chips (e.g., ML-based chip design). In some embodiments, ML-based chip design may include ML-based "chip floorplanning," or designing the physical layout of a computer chip. In some cases, this process may include arranging a hypergraph of circuit components (e.g., macros (memory components) and standard cells (logic gates, e.g., NAND, NOR, XOR, etc.)) on a chip canvas (e.g., a two-dimensional grid) such that performance metrics (e.g., power consumption, timing, area, and wire length) can be optimized while adhering to strict constraints on density and routing congestion. In some embodiments, ML-based chip design may include developing reinforcement learning (RL) techniques by framing the task as a sequential Markov decision process (MDP). In some cases, this process may include: ● State: The hypergraph (as an adjacency matrix), node features (width, height, type), edge features (number of connections), the current node (macro) to be placed, and the netlist graph metadata (routing assignments, total number of wires, macros, and standard cell clusters). ● Action: All possible positions (grid cells of the chip canvas) where the current macro can be placed, subject to predefined strict constraints on density or occlusion. ● Transition: Probability distribution over next states given a state and an action. ● Reward: The reward is the negative weighted sum of pro13 wire length, congestion, and density, and is always zero except for the last action.

[0057] In some cases, graph convolutional NNs may be used to learn feature embeddings for macros and other hypergraph components. In some instances, this architecture may offer advantageous geometric properties for this design space. Benchmark results of this approach show that it can produce chip floorplans that are comparable or better than human experts in under six hours, compared to months of manual human effort. This may enable AI-driven knowledge augmentation, where artificial agents can approach chip placement experience problems whose size and diversity may be on a scale larger than any human expert.

[0058] In further embodiments, the process of computer chip development may include automation and human-machine teaming. In some cases, this can potentially optimize and accelerate the process, including additional ways in which simulation AI can advance chip design. In some examples, the SimNet platform of NN solvers can be applied to the problem of FPGA heat sink design optimization, which is a multiphysics problem that can be approached with multiple parallel ML surrogates (e.g., one NN for the flow field and another NN for the temperature field).

[0059] Computer Architecture The "hardware-software" stack referred to in this disclosure does not, in some embodiments, include both hardware and software. Rather, the hardware-software stack may describe the "vertical" dimension that is currently suited for automatic differentiation as described herein (e.g., as opposed to the current paradigm where differentiation may only be possible "horizontally"). For example, the present invention enables automatic differentiation between the OS and applications, which spans multiple levels but is entirely in the software section of the hardware-software stack (FIG. 4). In some examples, the present disclosure further enables automatic differentiation between execution hardware, the OS, and libraries.

[0060] Computer architecture, as generally described herein, refers to the science and art of selecting and / or interconnecting hardware components to create a computer that meets any one of functional, performance, or cost goals. A non-limiting example of a computer architecture is shown in FIG. 5. Hardware in the present disclosure includes lower level components such as circuitry and logic. With reference to FIG. 5, the present disclosure enables automatic differentiation of a layer (510) and above (e.g., one or more of those layers / components).

[0061] The computer architecture of the present disclosure includes an abstract virtual machine (VM). The VM may be used to create relevant architectural differences and / or shared layers that run on various underlying computing platforms. Such a VM may have the ability to mold its own architecture (e.g., similar to how modern machine learning techniques learn from data). Thus, the VM may include a reconfigurable architecture. In some examples, the VM includes a parameterized description of the VM. In some examples, the VM may be optimized based on the size of memory, the number and width of registers, the available instruction set, instruction encoding, firmware and / or I / O implementation. In some examples, the optimization of the parameters of the VM described herein is achieved by endowing the VM with automatic differentiation capabilities.

[0062] In some examples, the VM may include an autodiff capability. Autodiff includes a native component at the operating system (OS) abstraction level. The autodiff capability may generally refer to emitting gradient programs in an intermediate representation (IR) and / or instruction set. As an example, the system may emit gradient programs in an intermediate representation (IR) and / or instruction set instead of the original source language. Thus, the system is capable of differentiating programs in various languages ​​for any program (e.g., AI / ML, data, and standard software applications) and providing inherent optimizations at the hardware-software interface.

[0063] This approach to VMs can be scaled by defining an n-dimensional grid of VMs with a flexible and adaptive network topology, as shown in FIG. 6. In some examples, each VM is an abstraction of a computation engine (e.g., CPU, GPU core, etc.). In some examples, each VM has private resources (e.g., not directly available to others). In some examples, the private resources comprise, as non-limiting examples, any one of a processor, a flexible instruction set (e.g., executed by the processor), an operation system (e.g., for resource allocation), private registers, private memory, input / output (I / O), or any combination thereof. In some examples, the processor communicates with any one of a private register, private memory, input / output (I / O), or any combination thereof. In some examples, it may be important for the system to be constrained, otherwise complete connectivity may make the problem difficult. In some examples, the VMs may exchange information through a special I / O register interface. In some examples, data may be passed using global absolute addressing over a global bus, or a neighbor-relative addressing mode. In some examples, all software that runs on the VMs is compiled using a compiler with automatic differentiation, such as those described herein. In some examples, a global bus is a way to communicate between a grid of VMs and some external interface (e.g., for writing / reading data) or other clusters of VMs.

[0064] In some examples, advantages of using a VM may include any one of: an isolated runtime environment; tunable low-level knobs; reconfigurability at runtime; fully observable; no hardware faults caused by incorrectly generated programs; easily portable; tight integration with a custom LLVM backend and automatic differentiation; internal firmware involved in tasks such as scheduling or memory management is also subject to automatic differentiation and optimization, or any combination thereof.

[0065] An illustration showing a possible flow of gradients from program outputs back to private VM resources is provided in Fig. 7, where the darkness of the gradient flow corresponds to the magnitude of the gradient (e.g., darker arrows correspond to larger gradients).

[0066] In some instances, one or more VMs utilized to deploy a production-grade system are optimized for various test applications. In some instances, the system is optimized from scratch for a specific test application. In some cases, the clear criteria for the usefulness of this system is that it a) demonstrates online adaptation and / or b) performs computations in a more efficient manner than the best "fixed" architectures can. In some instances, a successful system with one or more VMs, as described herein, provides a blueprint and proof of concept for future run-time reconfigurable systems. In some instances, this can open the door to a theory of computation that allows going beyond Turing-completeness and / or solving the so-called "halting problem."

[0067] In some examples, one or more VMs are implemented as a very thin layer written in portable C code. In some examples, one or more VMs are implemented on a standard CPU and / or GPU (e.g., x86_64 Intel® i7, Apple® M1, embedded ARM cores, even GPUs (CUDA), etc.). Thus, in some cases, the software may run on any machine that supports ANSI C. In some examples, one or more VMs are implemented on FPGAs, ASICs, NVMe, microcontrollers, AI-accelerators (e.g., Google-TPU®, Graphcore®, Cerebras®, SambaNova®, etc.), or any combination thereof.

[0068] In some examples, the VM is about 100 to about 500 lines of code. In some examples, the VM is about 300 lines of code. In some examples, the VM is about 100-200, 100-300, 100-400, 100-500, 200-300, 200-400, 200-500, 300-400, 300-500, or 400-500 lines of code. In some examples, the VM is about 100, 200, 300, 400, or 500 lines of code. In some examples, the VM is at least about 100, 200, 300, 400, or 500 lines of code. In some examples, the VM is up to about 100, 200, 300, 400, or 500 lines of code.

[0069] One or more VMs (e.g., entire networks of VMs) can be very generic, and any computing system can be emulated in such a way. As a result, in some examples, this provides a highly scalable and portable solution. In some examples, applications of the systems described herein leverage this versatility of VMs while also including benchmarking on various heterogeneous architectures, including various models of GPUs and TPUs. In some examples, the various heterogeneous architectures are in both distributed (e.g., multi- and hybrid cloud) settings and / or local (e.g., on-prem) settings.

[0070] Programming Languages ​​and Compilers In some embodiments, with the end of Moore's Law and Dennard scaling fast approaching, a diverse set of heterogeneous architectures are embraced as a way to continue to enable the performance demanded by data-intensive AI / ML, simulation, and scientific computing systems. In some examples, code bases are written with a given target in mind, which may mean that new AI / ML systems need to be created to handle new architectures or languages, in addition to presenting problems for users to understand and rewrite the code. Due to increasing problem complexity and hardware limitations, the growing size of hand-optimized libraries may not be able to keep up with future demands. Compounding this problem, compiler complexity grows with the complexity of computer architectures and workloads. In some examples, currently, to get the best performance for applications running on GPUs (e.g., AI / ML and simulation) and other types of accelerators, the best option is to use a specialized compiler tuned to the given accelerator.

[0071] Thus, the present disclosure is a system that operates across hardware architectures and languages ​​in a single framework by operating within a compiler. In some examples, the compiler is configured to compile software to run on one or more VMs. In some examples, the compiler is configured or capable of autodiff. In some examples, autodiffing includes translating a programming language into machine code that is processed on heterogeneous hardware, such as those described herein. More specifically, the autodiffing functionality includes enabling the translation of any software program (and system) in any programming language into machine code. Non-limiting examples of programming languages ​​include Python®, JavaScript®, Java®, C#, C, C++, Objective-C®, GO®, R®, Swift®, PHP®, Fortran®, APL®, Prolog®, SQL®, Matlab®, Julia®, Ada®, ALGOL®, Lisp®, Dart®, Ruby®, Rust®, etc. Additionally, transformations enabled by automatic differentiation can be processed on non-specific hardware, including combinations of heterogeneous hardware.

[0072] In some examples, a multi-level intermediate representation (MLIR) is analyzed as a possible replacement for the low-level intermediate representation (LLVM IR) used by the LLVM compiler framework (FIG. 8). In some examples, the MLIR is a target layer for automatic differentiation. In some embodiments, the objective of the system is to include a generic solution that is independent of the (LLVM) front end. In some embodiments, automatic differentiation is added as a native component of the compiler.

[0073] In some examples, initial validation of a compiler with automatic differentiation includes tools that take any existing code as LLVM IR and compute the derivatives (and gradients) of its functions. As an example, this can allow developers to use Enzyme to automatically create gradients of their source code without much additional work. In one such example, by working at the LLVM level, Enzyme can differentiate programs in various languages ​​(C, C++, Swift, Julia, Rust, Fortran, TensorFlow, etc.) with a single tool and achieve high performance by integrating with LLVM's optimization pipeline.

[0074] While tools exist for doing automatic differentiation in C++, including, for example, Adept or autodiff, these tools are frameworks that require a complete rewrite of the software to use library-specific calls. Furthermore, a direct one-to-one mapping is not always guaranteed in these frameworks. Furthermore, rewriting an entire software project can be resource-intensive, time-consuming, and / or error-prone. In some instances, having a high-performance automatic differentiation implementation in a programming language, e.g., C++, can make the difference between having a good solution for, for example, uncertainty estimation, and abandoning it due to computational complexity. In some instances, this is a bottleneck, prevalent in AI / ML (especially robotics) as well as large-scale distributed computing. In some instances, this bottleneck extends to similar scenarios, such as the inability to efficiently perform automatic differentiation at different levels of the stack and with different (or mixed) programming languages, which can severely constrain what a user can think of.

[0075] The ability to have gradient-based information in the compiler allows for several possible advances in the field of programming languages. The first possible advantage involves probabilistic programming. The probabilistic programming paradigm is equivalent to a probabilistic generative model with an executable program. Probabilistic programming languages ​​(PPLs) allow experts to leverage the power of programming languages ​​to create rich and complex models, while relying on a built-in inference backend that operates on any model written in the language, as previously described herein. In some instances, the use of PPLs may be undermined by computational costs. In some instances, efficient PPLs (e.g., Infer.net, BUGS, Edward, etc.) constrain the scope of modeling and inference to implement narrow, often hardware-specific PPLs.

[0076] A second possible advantage involves Domain Specific Languages ​​(DSLs). The use of DSLs allows to operate close to the "metal" and can be further optimized with hardware-aware gradients. As an example, as a proof of concept, the DSL "GraphIt®" can leverage machine programming to compute on graphs of various sizes and structures on various underlying hardware. As a further example, in the field of "intentional programming", the Halide programming language (not a DSL embedded in C++) was designed with two main components for two different types of programmers: (i) an algorithmic DSL for domain experts, and (ii) a scheduling component for programmers with a deep understanding of building optimal software for a given hardware target.

[0077] A third possible advantage involves differentiable programming (DP). DP is a programming paradigm in which the derivatives of a program are automatically calculated and, as previously described herein, is used in gradient-based optimization to adjust the program to achieve a given objective. DP can be used in a wide variety of fields, particularly scientific computing and ML. However, languages ​​are often designed for a single layer of a software-hardware stack. In some cases, a relatively simple syntax and set of operators are developed for full-stack DP for the systems of the present disclosure.

[0078] In some embodiments, auto-differentiation native compilers are used to overcome the challenges of system heterogeneity. In some examples, challenges include understanding interoperability across applications and devices, their semantics in the presence of extreme variation in abstractions, and / or how semantics can be engineered within programming languages ​​and compilers to obtain scalability and correctness across the full hardware / software stack.

[0079] In some examples, considerations for obtaining scalability include the complexities of parallel and distributed computing software, different computing architectures (e.g., ISA, cache, SIMD, etc.), and the complexities of memory hierarchies, network topologies, and / or heterogeneity. However, by using one or more VMs, scalability may be simplified. As an example, scalability may be achieved by creating a virtual machine-based abstraction for any computation and data movement to execute models such as those described herein. In some examples, a virtual machine (VM) models a single computational element. In some examples, communication between VMs is abstracted. In some examples, no assumptions are made about the characteristics of the system, including, but not limited to, the size, shape, and / or architecture of the system. In some examples, such characteristics of the system are discovered online, and thus the system may be scaled up at will. In some examples, power consumption includes a limiting factor. However, optimization of the system itself can ensure that tasks need to be performed as quickly as possible using as few resources as possible. In some examples, this leads to lower power consumption.

[0080] In some examples, considerations for correctness include the low-level implementation of the VM itself, the emulator engine, and glue code involved in "clock ticks." In some examples, the VM is implemented on any of the heterogeneous architectures described herein. In some examples, whether the emulator engine is implemented correctly is evaluated through one or more self-tests that can test any combination of instructions, perform memory and register moves, and / or arithmetic operations, and finally generate a checksum.

[0081] In some examples, considerations for obtaining accuracy include higher level functionality of the system. The disclosure described herein provides methods and systems for automatically generating computer programs. However, it may be difficult to determine whether a program that is generated and has never been run before is correct. In some examples, the systems described herein include a validation mechanism. In some examples, the validation mechanism takes a program as input and returns the probability that its execution will finish and / or the probability that the output is reliable. In some examples, the validation mechanism is implemented to provide the system with a program for modeling uncertainty. In some examples, the program is generated to have an inherent stochasticity, and the distribution of the program can be estimated. In some examples, the program includes a similar approach to the Variational Auto-Encoder (e.g., it is possible to determine whether a sample is out of distribution). In some examples, once the program is generated, a ground truth can be obtained. In some examples, if the output is undesirable, it serves as a negative sample. In some examples, this approach involves continuous data generation (positive samples / negative samples), validation, and / or adaptation as a way to ensure that the data used to train the system is accurate.

[0082] In some instances, operating at a lower level enabled by automatic differentiation allows for the manipulation and handling of lower level primitives (e.g., vector instructions, memory, I / O, etc.) As an example, some Julia automatic differentiation tools do not handle mutable memory and do not vectorize well, but Enzyme can accommodate the developments described herein.

[0083] In some examples, the LLVM infrastructure can be adopted to enable the use of any existing and future front-ends.

[0084] In some instances, new syntax is developed to support automatic differentiation and / or online learning. In some instances, this may yield contributions in the LLVM-IR layer and the LLVM backend.

[0085] In some examples, any hardware may be employed with the use of VMs as the core abstraction. In some examples, this removes most of the problems of programming heterogeneous machines. As an example, for a system with 4 GPUs and 16 CPUs, about 1000 VM units may run on the GPUs and about 500 VM units may run on the CPUs. In some examples, about 10,000 VM units may run on the GPUs. In some examples, about 500 to about 20,000 VM units may run on the GPUs. In some examples, the range is from about 500 to about 1,000, from about 500 to about 1,500, from about 500 to about 2,000, from about 500 to about 5,000, from about 500 to about 8,000, from about 500 to about 10,000, from about 500 to about 15,000, from about 500 to about 20,000, from about 1,000 to about 1,500, from about 1,000 to about 2,000. , about 1,000 to about 5,000, about 1,000 to about 8,000, about 1,000 to about 10,000, about 1,000 to about 15,000, about 1,000 to about 20,000, about 1,500 to about 2,000, about 1,500 to about 5,000, about 1,500 to about 8,000, about 1,500 to about 10,000, about 1,50 0 to about 15,000, about 1,500 to about 20,000, about 2,000 to about 5,000, about 2,000 to about 8,000, about 2,000 to about 10,000, about 2,000 to about 15,000, about 2,000 to about 20,000, about 5,000 to about 8,000, about 5,000 to about 10,000, about 5,000 to about 1 5,000, about 5,000 to about 20,000, about 8,000 to about 10,000, about 8,000 to about 15,000, about 8,000 to about 20,000, about 10,000 to about 15,000, about 10,000 to about 20,000, or about 15,000 to about 20,000 VM units are executed on the GPU. In some examples, about 500, 1,000, 1,500, 2,000, 5,000, 8,000, 10,000, 15,000, or 20,000 VM units are executed on the GPU. In some examples, at least about 500, 1,000, 1,500, 2,000, 5,000, 8,000, 10,000, 15,000, or 20,000 VM units run on the GPU.In some examples, at most about 500, 1,000, 1,500, 2,000, 5,000, 8,000, 10,000, 15,000, or 20,000 VM units run on the GPU. In some examples, about 100 to about 1,000 VM units may run on the CPU. In some examples, about 100 to 500, 100 to 700, 100 to 1000, 500 to 700, 500 to 1,000, or 700 to 1,000 VM units run on the CPU. In some examples, about 100, 200, 300, 400, 500, 600, 700, 800, 900, or 1,000 VM units run on the CPU. In some examples, at least about 100, 200, 300, 400, 500, 600, 700, 800, 900, or 1,000 VM units execute on the CPU. In some examples, at most about 100, 200, 300, 400, 500, 600, 700, 800, 900, or 1,000 VM units execute on the CPU.

[0086] DP describes a powerful concept in deep learning and includes parameterized software modules that can be trained with some form of gradient-based optimization. The automatic differentiation techniques described herein (e.g., compilers, OS, etc.) can have similar impacts, except for the complete software-hardware stack. That is, as non-limiting examples, hardware-software codeine, scheduling (compiler instructions, jobs in single-core and multiprocessor settings, etc.), resource management and task allocation, or gradient-based optimization for large-scale and heterogeneous computing (including petascale and exoscale systems).

[0087] Compared to the majority of automatic differentiation tools that differentiate programs at compile time (e.g., Fortran, C, etc.), Enzyme is based on the LLVM compiler instead of a framework specialized for automatic differentiation, and can emit gradient programs in LLVM IR instead of the original source language. This approach can allow Enzyme to benefit from the language support, optimizations, and / or maturity of the LLVM platform.

[0088] In some examples, GPU kernels can be generated by an LLVM compiler, so Enzyme includes the first fully automatic reverse-mode automatic differentiation tool for generating gradients for GPU kernels. As an example, the system can run Enzyme for HPC applications on NVIDIA and AMD GPUs with <20x runtime overhead. In some examples, Enzyme can be evolved to improve by at least an order of magnitude. In some examples, Enzyme can be applied (and proven) to more heterogeneous computing platforms.

[0089] Systems for AI / ML, Simulation, Scientific Computing and Data-Driven Technologies The systems referred to herein may comprise a dynamic combination of software, hardware, data, algorithms, and / or people. The challenges associated with systems for AI / ML, simulation, scientific computing, and data-driven techniques may span across domains and across the entire stack. As an example, the proliferation of heterogeneous hardware and domain-specific libraries / frameworks makes it nontrivial to build (let alone optimize and / or maintain) a system with general-purpose tools or algorithms. In some cases, optimization across components is becoming increasingly important to achieve maintainable and adaptive scalability across the entire hardware-software stack.

[0090] Furthermore, computing environments are increasingly distributed and may inherently be at risk for security issues, such as edge computing and public clouds (very common in scientific research environments dealing with sensitive data sets). In some embodiments, these safety issues motivate the development, validation, and / or communication of guiding principles for large-scale system design. In some cases, novel approaches to systems disclosed herein comprise (a) machine-programmable pipelines for automatically generating optimal configurations of system components, and / or (b) systems engineering for AI / ML and simulation techniques. The machine-programmable pipelines in (a) can be rigorously enabled by the novelty of multi-layer differentiability of the hardware-software stack. The systems engineering in (b) includes frameworks that can be implemented for simulation intelligence and have been proven to enable the development of reliable and scalable learning / simulation techniques.

[0091] In some embodiments, in simulation intelligence and / or computer systems in general, guiding principles for system development include how to build reliable systems with unreliable components. Non-limiting examples of unreliable components include noisy or faulty sensors, human error and AI error, etc. Thus, there may be great value in quantifying uncertainties and propagating them throughout the system to arrive at a concept or measure of reliability. In some examples, principled uncertainty propagation is valid between modules, up and down abstraction levels. With the full-stack automatic differentiation methods described herein, this local and global type of uncertainty propagation may be possible. At the abstraction level of the hardware-software stack described herein, uncertainty-aware methods and corresponding metrics may be provided. In this way, the hardware-software stack described herein may provide a theoretically grounded and empirically validated measure of reliability of AI systems.

[0092] In some instances, ongoing changes in hardware platforms (and combination with a wide range of sensors, data streams, etc.) can raise concerns about maintaining accuracy, robustness, and precision. As one example, programs can often give unexpected results on different platforms due to differences in hardware semantics (e.g., various memory types and memory consistency models, etc.). Quantifying uncertainty at all levels of the hardware-software stack, including information flow between levels, can provide a principled real-time uncertainty measure for heterogeneous systems and essentially all scales.

[0093] In some examples, in addition to the machine programming, computer architectures, programming languages ​​and compilers, and systems described herein, substantial advances may also be made in the context of high performance computing (HPC). In some examples, advances may be by application areas encompassing simulation systems and scientific computing.

[0094] Target Applications and Systems In some examples, the target applications and systems described herein comprise those that expose the problems described herein. In some examples, the target applications and systems described herein include valuable real-world applications of the technology. In some examples, the target applications and systems described herein are diverse in many ways, such as data types, computing platforms, and deployment settings and their efficiency and sensitivity requirements. In some examples, the target applications are comprised of open source code bases that can be executed with the systems and methods described herein. In some examples, such open source code bases can be used for benchmarking the systems and methods described herein.

[0095] Exemplary target applications and corresponding systems include Earth system modeling. In some examples, Earth system modeling includes open source code bases such as the "versatile ocean simulator" ("Veros®"), which aims to enable high performance ocean modeling with a clear focus on flexibility and usability. Veros features a full-fledged primitive equation ocean model that supports everything between idealized toy models and realistic high resolution global ocean simulations. Veros can support a NumPy backend for small problems, as well as a high performance JAX backend with CPU and GPU support. It is fully parallelized via MPI and supports distributed execution on any number of nodes, including multi-GPU architectures. The dynamic core of Veros is based on pyOM2, an ocean model with a Fortran backend and a Fortran frontend and a Python frontend. The open source Veros project also includes benchmarks that can be used directly and reused for our methods and systems (and engineering science in general).

[0096] In addition to this ocean simulator, several complementary SI systems can be deployed. In some examples, the complementary SI system comprises modeling large-scale simulation pipelines for atmospheric transport of pollutants, nuclear winter simulation, and / or space weather. For atmospheric transport, the challenge involves running HPC physics-based numerical simulations on cost-effective hardware. In some examples, ML-based surrogate modeling techniques are utilized to automatically identify computational bottlenecks and replace them with data-driven surrogates that are more efficient in scale. In some examples, MP-based methods are utilized. In some examples, the complementary SI system comprises heterogeneous software-hardware stack components, various programming languages ​​(e.g., Julia, Python, C++, Fortran, etc.), and deployment settings involving HPC (e.g., both hybrid cloud and multi-cloud setups).

[0097] Exemplary target applications and corresponding systems include energy grid simulation and optimization. In some examples, open source modeling and simulation systems for energy grids provide testbeds for the methods and systems described herein. Such open source systems include the Breakthrough Energy Open Source Energy System Model and the DOE Grid Optimization Competition. In some examples, the latter can further provide benchmarking data sets and results for evaluating new classes of methods developed as described herein.

[0098] Exemplary target applications and corresponding systems include machine learning optimization of experimental design. ML algorithms can be used to design experiments such that the results are as informative as possible about the underlying processes. Outer-loop algorithms are often used, but valuable information may be lost in experimental settings that are more complex and error-prone than simple testbeds. In some instances, optimizing ML algorithms is not trivial, especially in heterogeneous experimental platforms. In some instances, the system-wide differentiability described herein is useful for ML-based optimization. As an example, forward looking use-cases involving simulation intelligence include "biotech laboratory infrastructure" and "cloud labs" (e.g., Emerald Cloud Lab®), which may also require machine programming and automatic differentiation techniques.

[0099] An exemplary target application and corresponding system includes AI-driven sensor design. AI-enabled sensor design involves the use of inverse design and related machine learning techniques to optimally design data acquisition hardware with respect to user-defined cost functions or design constraints. Such an approach can dramatically improve the overall performance of the sensor and the broader data ingestion / modeling system. In some instances, this may be achieved through non-intuitive design choices. As an example, soil (and carbon) sensor data from Microsoft's Climate group (as well as NASA) may be used in this target application and system.

[0100] Beyond the use of these applications described herein, operationalizing datasets, exercises, code bases, etc. towards new MP benchmarks is also important in developing the technology described herein. MP has been developed and evaluated primarily on typical use cases, e.g., with NLP benchmarks. Science and engineering are in many ways distinct from NLP and other mainstream ML, implying that there are many more challenges and opportunities to test, break, and improve MP within real-world applications (not to mention unknown unknowns to discover).

[0101] MP benchmarks may have multiple levels, each measuring slightly different characteristics, but when a single number is used, the default level A numbers may be compared. In some examples, a class of MP benchmarks involves (A) giving a sufficiently large number of example <input, output> pairs to infer the program that best explains the data, and using this program to obtain test set accuracy. The <input, output> pairs are generated using a "hidden" program drawn from some parameterized family of algorithms. In addition to choosing correct programs over incorrect programs, additional score may be awarded for faster execution (when two or more programs produce the same correct result). In some examples, a class of MP benchmarks involves (B) similar to A, but using raw<input,output> In addition to providing pairs, additional inputs, either text (copilot-style comments guiding the system) or images, are possible for each sample. This measures whether a learned program can quickly adapt to new tasks. For example, one might expect a program to learn to sort regardless of the data type (integers or strings). The data type information could be that additional information. In some examples, the MP benchmark class includes (C) Model Self-Evaluation. Given a prompt, the system is asked to generate a program and is given an estimate of the uncertainty of this solution. In other words, it is evaluated whether the system is "very confident". In some examples, this is done by measuring the cross-entropy between the ground truth and the prediction. In some examples, the MP benchmark class includes (D) predicting the output of an <input, program> pair. This may require effectively solving a task such as the halting problem. Tasks C and D can be combined to generate both a prediction as well as a confidence level.

[0102] Provided herein are techniques that can efficiently, dynamically, and reliably utilize heterogeneous hardware accelerators. This may entail an architectural platform for scaling existing systems and supporting new systems. Provided herein is a class (or paradigm) of methods (e.g., the SI-MP "flywheel") for innovating in this field, but also a fundamental requirement to ensure this direction of innovation: MP benchmarks. Such MP benchmarks can conclusively evaluate scalability in a variety of real-world empirical challenges that utilize heterogeneous combinations across the hardware-software stack. The present disclosure explores all levels of the hardware-software stack, while facilitating improved reasoning about important system characteristics such as accuracy and precision. To this end, scalability is also evaluated end-to-end with respect to the complete hardware-software stack (see Example 1). In some examples, innovations and changes in certain parts of the hardware-software stack may have effects that propagate throughout the entire system (and perhaps in hidden ways covered by abstractions).

[0103] The Earth system and energy grid applications presented herein provide examples of extreme-scale modeling and simulation applications. In some examples, extreme-scale modeling and simulation applications require modeling complex phenomena more holistically across multiple scales and physics, requiring high-resolution dynamically coupled simulation workflows. In some examples, these applications integrate data analytics and new AI / ML approaches, both at scale. Thus, the complexity and scale of the resulting SI workflows may present new challenges and requirements for future advanced computing ecosystems. In some examples, the integration of coupled models, data, and analytics requires a composable execution environment, which integrates appropriate resources and capabilities, as well as is directly addressed within the present disclosure with general-purpose, scalable technology.

[0104] Artificial Intelligence (AI) and Machine Learning (ML) Methods The disclosed system may comprise a platform for or built using one or more AI / ML approaches. AI approaches generally include the intelligence demonstrated by machines that is typically associated with intelligent humans. In some embodiments, the AI ​​includes probabilistic techniques (e.g., Bayesian networks, Hidden Markov models, Kalman filters, Particle filters, Decision theory, Utility theory, etc.). In some embodiments, the AI ​​includes logic such as, but not limited to, propositional logic, first-order logic, fuzzy logic, default logic, non-monotonic logics, description logic, situation calculus, event calculus, fluent calculus, causal calculus, belief calculus, modal logics, and paraconsistent logic. In some embodiments, the AI ​​includes search and optimization algorithms (e.g., random optimization, beam search, metaheuristics such as simulated annealing, etc.). In some embodiments, the search and optimization algorithms include evolutionary algorithms, including, but not limited to, genetic algorithms, gene expression programming, and genetic programming. In some embodiments, the AI ​​includes symbolic AI that is based on high-level symbolic (e.g., human-readable) representations of problems, logic, and searches.In some embodiments, the AI ​​includes a specialized language (e.g., Lisp, Prolog, TensorFlow, etc.). In some embodiments, the AI ​​includes natural language processing (NLP). In some embodiments, the AI ​​includes specialized hardware (e.g., AI accelerators, neuromorphic computing, etc.). In some embodiments, the AI ​​includes an ML approach including a classifier and / or statistical learning. In some embodiments, the AI ​​includes an ML approach including an artificial neural network (e.g., a feedforward neural network, a recurrent neural network, etc.). In some embodiments, the AI ​​includes an ML approach including deep learning.

[0105] In some examples, the one or more machine learning (ML) approaches are supervised, semi-supervised, or unsupervised. In some examples, the one or more ML techniques perform classification or clustering. In some examples, the machine learning techniques comprise classical machine learning methods, such as, but not limited to, Support Vector Machines (SVMs) (e.g., one-class SVMs), k-nearest neighbors (KNNs), isolation forests, random forests, or any combination thereof. In some examples, the machine learning techniques comprise deep leaning methods (e.g., deep neural networks (DNNs)), such as, but not limited to, Convolutional Neural Networks (CNNs) (e.g., one-class CNNs), Recurrent Neural Networks (RNNs), Transformers, Graph Neural Networks (GNNs), Convolutional Graph Neural Networks (CGNNs), or any combination thereof.

[0106] In some embodiments, the classical ML method includes one or more algorithms that learn from existing observations (e.g., selected features) to predict an output. In some embodiments, the one or more algorithms perform clustering of the data. In some examples, the classical ML algorithm for clustering includes K-means clustering, mean-shift clustering, density-based spatial clustering of applications with noise (DBSCAN), expectation-maximization (EM) clustering (e.g., using Gaussian mixture models (GMM)), agglomerative hierarchical clustering, or any combination thereof. In some embodiments, the one or more algorithms perform classification of the data. In some examples, the classical ML algorithm for classification includes logistic regression, naive Bayes, KNN, random forest, separation forest, decision tree, gradient boosting, support vector machine (SVM), or any combination thereof. In some examples, the SVM includes one-class SMV or multi-class SVM.

[0107] In some embodiments, the deep learning method comprises one or more algorithms that learn by extracting new features to predict outputs. In some embodiments, the deep learning method comprises one or more layers. In some embodiments, the deep learning method comprises a neural network (e.g., a DNN comprising multiple layers). A neural network generally comprises connected nodes in a network that can perform functions such as transforming or converting input data. In some embodiments, an output from a given node is passed as an input to another node. A node in a network generally comprises an input unit, a hidden unit, an output unit, or a combination thereof. In some embodiments, an input node is connected to one or more hidden units. In some embodiments, one or more hidden units are connected to an output unit. A node generally can take an input through an input unit and generate an output from an output unit using an activation function. In some embodiments, an input or output comprises a tensor, a matrix, a vector, an array, or a scalar. In some embodiments, the activation function is a binary step activation function, a linear activation function, a rectified linear unit (ReLU) activation function, a leaky ReLU activation function, a parameterized ReLU, a sigmoid activation function, a hyperbolic tangent activation function, a Softmax activation function, an exponential linear unit, or a Swiss activation function.

[0108] The connections between the nodes further comprise weights for adjusting the input data to a given node (e.g., activating the input data or deactivating the input data). In some embodiments, the weights are learned by a neural network. In some embodiments, the neural network is trained to learn the weights using gradient-based optimization. In some embodiments, the gradient-based optimization includes one or more loss functions (e.g., Binary Crossentropy, Binary Focal Crossentropy, Categorical Crossentropy, Categorical Hinge, Cosine Similarity, Hinge, Huber, KL Divergence, LogCosh, Mean Absolute Error, Mean Absolute Percentage Error, Mean Squared Error, Mean Squared Logarithmic Error, Poisson, Reduction, Sparse Categorical Crossentropy, Squared Hinge, etc.). In some embodiments, the gradient-based optimization is gradient descent, conjugate gradient descent, stochastic gradient descent, or any variant thereof (e.g., adaptive moment estimation (Adam)). In some further embodiments, the gradients in the gradient-based optimization are calculated using backpropagation. In some embodiments, the nodes are organized into a graph to generate a network (e.g., a graph neural network).In some embodiments, the nodes are organized into one or more layers to generate a network (e.g., a feedforward neural network, a convolutional neural network (CNN), a recurrent neural network (RNN), etc.). In some embodiments, the CNN includes a one-class CNN or a multi-class CNN.

[0109] In some embodiments, the neural network comprises an autoencoder. In some embodiments, the autoencoder comprises an encoder and a decoder. In some embodiments, the encoder reduces the dimensionality of the input and the decoder regenerates the input. Thus, the encoding process is verified and improved by attempting to regenerate the input. In some embodiments, the autoencoder may learn a representation of the dataset through training that ignores unimportant parts of the dataset. In some embodiments, the autoencoder includes a regularized autoencoder, a concrete autoencoder, or a variational autoencoder (VAE). In some embodiments, the autoencoder is used for dimensionality reduction, information retrieval, anomaly detection, image processing, drug discovery, popularity prediction, or machine translation.

[0110] In some embodiments, the neural network comprises a generative adversarial network (GAN). In some embodiments, the GAN comprises at least two neural networks. In some embodiments, one of the at least two neural networks comprises a generative network. In some embodiments, the generative network comprises a statistical model of a joint probability distribution P(X,Y) given observed variables X and target variables Y. In some embodiments, one of the at least two neural networks comprises a discriminative network. In some embodiments, the discriminative network comprises a statistical model of a conditional probability P(Y|X=x) of target variable Y given observed values ​​x. In some embodiments, the generative network learns to map from a latent space to a data distribution of interest, and the discriminative network distinguishes candidates generated by the generator from the true data distribution.

[0111] In some embodiments, the neural network comprises one or more regression layers. In some embodiments, the one or more regression layers are one or more long short-term memory (LSTM) layers or gated recurrent units (GRUs). In some embodiments, the one or more regression layers perform sequential data classification and clustering, where data ordering (e.g., time series data) is taken into account. In such embodiments, future predictions are made by the one or more regression layers according to the sequence of past events. In some embodiments, the regression layers retain or "remember" important information while selectively "forgetting" what is not essential for classification.

[0112] In some embodiments, the neural network includes one or more convolutional layers. In some embodiments, the inputs and outputs are tensors that represent variables or attributes in a dataset (e.g., features), which may be referred to as feature maps (or activation maps). In such embodiments, the one or more convolutional layers are referred to as a feature extraction phase. In some embodiments, the convolution is a one-dimensional (1D) convolution, a two-dimensional (2D) convolution, a three-dimensional (3D) convolution, or any combination thereof. In further embodiments, the convolution is a 1D transposed convolution, a 2D transposed convolution, a 3D transposed convolution, or any combination thereof.

[0113] In some embodiments, the neural network comprises one or more attention layers. An attention layer may generally emphasize some parts of a dataset and attenuate others. Thus, attention layers may be used to process or analyze various types of data, such as language or sensory data (e.g., sound, images, video, text, etc.). In some embodiments, the attention layer comprises a function that can take a representation of an element and map it to a scalar value called an "attention weight." An attention weight may generally emphasize how a machine learning model or layer therein focuses on or "weights" features in a dataset. In some embodiments, the attention layer is between one or more layers of a neural network, such as those described herein. In some embodiments, the neural network with one or more attention layers comprises a Transformer, a self-attention neural network, a multi-head attention neural network, or a gated attention neural network.

[0114] The layers in the neural network may further include one or more pooling layers before or after the convolutional layers. In some embodiments, the one or more pooling layers remove the dimensionality of the feature maps using filters that summarize regions of the matrix. In some embodiments, this reduction reduces the number of output samples and therefore the parameters and computational resources available to the neural network. In some embodiments, the one or more pooling layers include max pooling, min pooling, average pooling, global pooling, norm pooling, or a combination thereof. In some embodiments, max pooling reduces the dimensionality of the data by taking the maximum value in a region of the matrix. In some embodiments, this helps to capture the one or more features that are most important. In some embodiments, the one or more pooling layers are one dimensional (1D), two dimensional (2D), three dimensional (3D), or any combination thereof.

[0115] The neural network may further comprise one or more flattening layers that may flatten the input passed to the next layer. In some embodiments, the input (e.g., feature map) is flattened by reducing the input to a one-dimensional array. In some embodiments, the flattened input may be used to output a classification of the object. In some embodiments, the classification comprises binary or multi-class classification of visual data (e.g., images, videos, etc.) or non-visual data (e.g., measurements, audio, text, etc.). In some embodiments, the classification comprises binary classification of images (e.g., cats or dogs). In some embodiments, the classification comprises multi-class classification of text (e.g., identifying handwritten digits). In some embodiments, the classification comprises binary classification of measurements. In some examples, the binary classification of measurements comprises classification of the performance of a system using physical measurements described herein (e.g., normal or abnormal, normal or abnormal).

[0116] The neural network may further comprise one or more dropout layers. In some embodiments, the dropout layers are used during training of the neural network (e.g., to perform binary or classification multi-class classification). In some embodiments, the one or more dropout layers randomly set some weights as 0 (e.g., about 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80% of the weights). In some embodiments, setting some weights to 0 also sets the corresponding elements in the feature map to 0. In some embodiments, the one or more dropout layers may be used to prevent the neural network from overfitting.

[0117] The neural network further comprises one or more dense layers comprising a fully connected network. In some embodiments, information is passed through the fully connected network to generate a predicted classification of the object. In some embodiments, an error associated with the predicted classification of the object is also calculated. In some embodiments, the error is backpropagated to improve the prediction. In some embodiments, the one or more dense layers include an activation function such as those described herein (e.g., a Softmax function). In some embodiments, the activation function converts a vector of numbers into a vector of probabilities. In some embodiments, these probabilities are then used for classification.

[0118] Computer Systems In one aspect, the present disclosure provides a computer system programmed or otherwise configured to perform any of the methods of the present disclosure, e.g., the subject methods for medical imaging. Figure 11 shows a computer system (1101) programmed or otherwise configured to perform the methods described herein. The computer system (1101) may be a user's electronic device or a computer system located remotely relative to the electronic device. The electronic device may be a mobile electronic device.

[0119] The computer system (1101) may include a central processing unit (CPU, also referred to herein as "processor" and "computer processor") (1105), which may be a single-core or multi-core processor, or multiple processors for parallel processing. The computer system (1101) also includes memory or memory locations (1110) (e.g., random access memory, read-only memory, flash memory), an electronic storage unit (1115) (e.g., hard disk), a communication interface (1120) (e.g., network adapter) for communicating with one or more other systems, and peripheral devices (1125), such as cache, other memory, data storage, and / or electronic display adapters. The memory (1110), storage unit (1115), interface (1120), and peripheral devices (1125) communicate with the CPU (1105) via a communication bus (solid lines), such as a motherboard. The storage unit (1115) may be a data storage unit (or data repository) for storing data. The computer system (1101) may be operatively coupled to a computer network ("network") (1130) using the communication interface (1120). The network (1130) may be the Internet, an Internet and / or an extranet, or an intranet and / or an extranet in communication with the Internet. The network (1130) may, in some cases, be a telecommunications and / or data network. The network (1130) may include one or more computer servers that may enable distributed computing, such as cloud computing. The network (1130) may, in some cases, implement a peer-to-peer network with the computer system (1101), which may enable devices coupled to the computer system (1101) to function as clients or servers.

[0120] The CPU (1105) may execute a sequence of machine-readable instructions, which may be embodied in a program or software. The instructions may be stored in a memory location, such as the memory (1110). The instructions may be directed to the CPU (1105), which may then program or otherwise configure the CPU (1105) to perform the methods of the present disclosure. Examples of operations performed by the CPU (1105) may include fetch, decode, execute, and writeback.

[0121] The CPU 1105 may be part of a circuit, such as an integrated circuit. One or more other components of the system 1101 may be included in the circuit. In some cases, the circuit is an application specific integrated circuit (ASIC).

[0122] The storage unit (1115) can store files such as drivers, libraries, and saved programs. The storage unit (1115) can store user data, such as user preferences and user programs. The computer system (1101) may include one or more additional data storage units, in some cases located outside the computer system (1101) (e.g., on a remote server in communication with the computer system (1101) through an intranet or the Internet).

[0123] The computer system (1101) can communicate with one or more remote computer systems via the network (1130). For example, the computer system (1101) can communicate with a remote computer system of a user (e.g., a subject, an end user, a consumer, a medical provider, an imaging technician, etc.). Examples of remote computer systems include a personal computer (e.g., a portable PC), a slate or tablet PC (e.g., an Apple® iPad, a Samsung® Galal 11 Tab), a phone, a smartphone (e.g., an Apple® iPhone, an Android-enabled device, a Blackberry®), or a personal digital assistant. A user can access the computer system (1101) via the network (1130).

[0124] The methods described herein may be implemented by machine (e.g., computer processor) executable code stored on an electronic storage location of the computer system (1101), such as, for example, memory (1110) or electronic storage unit (1115). The machine executable or machine readable code may be provided in the form of software. In use, the code may be executed by the processor (1105). In some cases, the code may be retrieved from the storage unit (1115) and stored in the memory (1110) for easy access by the processor (1105). In some situations, the electronic storage unit (1115) may be omitted and the machine executable instructions are stored in the memory (1110).

[0125] The code may be pre-compiled and configured for use with a machine having a processor adapted to execute the code, or may be compiled during run-time. The code may be provided in a programming language that may be selected to allow the code to be executed in a pre-compiled or as-compiled manner.

[0126] Aspects of the systems and methods provided herein, such as the computer system (1101), can be embodied in programming. Various aspects of the technology can be considered as "products" or "articles of manufacture," typically in the form of machine (or processor) executable code, and / or associated data carried on or embodied in some type of machine-readable medium. The machine-executable code can be stored in an electronic storage unit, such as a memory (e.g., read-only memory, random access memory, flash memory) or a hard disk. A "storage" type medium can include any or all of the tangible memory of a computer, a processor, or its associated modules, such as various semiconductor memories, tape drives, disk drives, etc., which may provide non-transitory storage at any time for software programming. All or portions of the software may be communicated from time to time over the Internet or various other telecommunications networks. Such communication may, for example, enable loading of the software from one computer or processor to another, for example, from a management server or host computer to the computer platform of an application server. Thus, another type of medium that may carry software elements includes optical, electrical, and electromagnetic waves, such as those used across physical interfaces between local devices, through wired and optical landline networks, and over various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., may also be considered software-bearing media. As used herein, unless limited to non-transitory, tangible "storage" media, terms such as computer or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.

[0127] Thus, a machine-readable medium such as a computer executable code may take many forms, including but not limited to tangible storage media, carrier wave media, or physical transmission media. For example, non-volatile storage media, including optical or magnetic disks, or any storage device in any computer, may be used to implement the databases shown in the figures. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables, copper wire and optical fibers, including the wires that make up a bus in a computer system. Carrier wave transmission media may take the form of electric or electromagnetic signals, or acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Thus, common forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, a DVD or DVD-ROM, any other optical medium, punch cards paper tape, any other physical storage medium with a pattern of holes, a RAM, a ROM, a PROM and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave carrying data or instructions, a cable or link carrying such a carrier wave, or any other medium from which a computer can read programming code and / or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0128] The computer system (1101) may comprise or communicate with an electronic display (1135) that comprises a user interface (UI) (1140) for providing a portal for interacting, for example, with one or more layers of a stack (e.g., FIG. 1 or FIG. 4) or within a computer architecture described herein (e.g., FIG. 5). In some embodiments, the portal may interact with one or more VMs as described herein. The portal may be provided through an application programming interface (API). A user or entity may also interact with various elements within the portal via the UI. Examples of UIs include, but are not limited to, graphical user interfaces (GUIs) and web-based user interfaces.

[0129] The methods and systems of the present disclosure may be implemented by one or more algorithms. The algorithm may be implemented by software when executed by the central processing unit (1105). For example, the algorithm may be configured to receive a data set or data stream, provide an AI-enabled simulation comprising the data set or data stream and one or more domain-specific computing modules, generate a future prediction based on the AI-enabled simulation, or a combination thereof. EXAMPLES

[0130] Example 1 - End-to-End Scalability Validation To validate our scalability claims, we performed a proof-of-concept of a grid of VMs. A small definition of the VM in C was compiled using standard GCC for a multicore CPU, as well as an NVIDIA GPU using the CUDA compiler. Preliminary results showed that the overhead caused by the VM emulator is roughly 5-6x. In other words, a 600MHz machine can be efficiently emulated on a 3GHz CPU. Without any modifications, 100.000 instances of the VM were run on a single Volta GPU, each running at about 200MHz. This implementation can be scaled to multiple heterogeneous nodes, provided that the GPU and CPU use the same memory space.

[0131] Example 2 - End-to-End Correctness Demonstration The small VMs were implemented on old 8-bit CPU architectures such as the 6502 or Intel 8080. To verify that the engine was implemented correctly, a series of self-tests were run to test all combinations of instructions, memory and register moves, arithmetic operations, and finally generate a checksum. This procedure was run for billions of cycles, which made implementation-related bugs less likely. A set of selected programs for the grid of VMs were then executed, and the memory contents were finally examined. An exemplary diagram of memory contents is shown in Figure 9, which illustrates the memory contents of a grid of 16 VMs that were running an 18-byte long program that generates a selected pattern. Note that all VMs operate asynchronously. This approach was also tested on a GPU implementation running 100,000 VMs simultaneously. It would be simple to dump the contents of all the VMs and then compare them to some reference output, which could allow for automation.

[0132] Example 3 - Heterogeneous Architecture Demonstration As described in Examples 1 and 2, the VM system was successfully tested on CPUs, GPUs, and various mixes of both. Because the VM is software-defined, it was clear that any Turing-complete machine could be implemented and emulated. In addition to the major CPU and NVIDIA GPU varieties, the same code as in Example 2 was tested on embedded CPUs using only ANSI C with no system libraries, OpenCL for Intel CPUs, and Intel / AMD GPUs.

[0133] We also evaluated the performance of an FPGA implementation of such VMs. We fit about 1000 cores into a medium-sized Xilinx ZYNQ 7045 FPGA running at 600 MHz. After applying this VM layer, all these different architectures could run the same code, which made this approach very flexible. The FPGA floorplan showing how many VMs were placed in the programmable logic is shown in Figure 10.

[0134] Example 4 - Scalability Test on a GPU Node The scalability of the GPU nodes was tested by measuring the total performance as instructions per second. This test showed the total number of VM cycles that could be executed. Each VM was a small 8-bit CPU with 4kB of memory and was tested on an RTX A6000. The results are shown in Table 2, where the performance is shown in million instructions per second.

[0135] [Table 2]

[0136] As shown, performance scales well with the number of VMs. Results also show that it is possible to run >10,000 parallel programs / core simultaneously.

[0137] Specific Definitions Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this subject matter belongs.

[0138] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Any reference to "or" herein is intended to include "and / or" unless specifically stated otherwise.

[0139] Throughout this specification, references to "some embodiments," "further embodiments," or "particular embodiments" mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment. Thus, appearances of the phrases "in some embodiments," or "in further embodiments," or "in a particular embodiment" in various places throughout this specification do not necessarily all refer to the same embodiment. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0140] The term "simulation intelligence" referred to herein generally describes the merger of artificial intelligence / machine learning (AI / ML), simulation, and scientific computing. In some cases, simulation intelligence research studies processes and systems in silico to better understand and discover in situ phenomena. In some examples, simulation intelligence serves as an in silico testbed for developing, experimenting, and / or testing AI / ML prior to in situ use. In some cases, simulation intelligence is enabled by a dedicated operating system (e.g., the Sim-AI OS described herein). In some cases, simulation is enabled by a system with capabilities such as automatic differentiation and / or machine programming.

[0141] Although preferred embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. The present disclosure is not intended to be limited by the specific examples provided herein. The present disclosure has been described with reference to the foregoing specification, and the description and illustration of the embodiments herein are not meant to be interpreted in a limiting sense. Numerous variations, changes, and substitutions will occur to those skilled in the art without departing from the present disclosure. Furthermore, it should be understood that all aspects of the present disclosure are not limited to the specific depictions, configurations, or relative proportions described herein, which depend upon a variety of conditions and variables. It should be understood that various alternatives to the embodiments of the present disclosure described herein may be employed in implementing the present disclosure. Thus, it is contemplated that the present disclosure shall also cover any such alternatives, modifications, variations, or equivalents. It is intended that the following claims define the scope of the present disclosure, and that methods and structures within the scope of these claims and their equivalents are covered thereby.

Claims

1. 1. A system comprising: one or more virtual machines (VMs) configured to generate a reconfigurable architecture, each of the one or more VMs including an abstraction of a computer engine; a compiler configured to compile software to be executed on the one or more VMs, the compiler being configured or capable of autodiffing; Including, the system.

2. The system of claim 1 , wherein the one or more VMs include an autodiff feature.

3. The system of claim 2 , wherein the autodiff function includes emitting gradient programs in an intermediate representation (IR) and / or an instruction set.

4. The system of claim 1 , wherein the one or more VMs are parameterized.

5. The system of claim 1 , wherein the one or more VMs have one or more parameters that are optimized.

6. 6. The system of claim 5, wherein the one or more parameters include a size of memory, a number and / or width of registers, an available instruction set, an instruction encoding, a firmware implementation, input / output (I / O), or any combination thereof.

7. The system of claim 1 , wherein each of the one or more VMs includes private resources.

8. The system of claim 1 , wherein the compiler comprises a multi-level intermediate representation (MLIR) compiler or a low-level intermediate representation (LLVM IR) compiler.

9. 2. The system of claim 1, wherein the autodiff comprises translating a programming language into machine code to be processed on heterogeneous hardware.

10. 10. The system of claim 1, wherein the compiler enables the use of probabilistic programming, domain-specific languages, differentiable programming, or any combination thereof.

11. 10. The system of claim 1, wherein the one or more VMs run on a GPU, a CPU, an FPGA, an ASIC, an NVMe, a microcontroller, an AI-accelerator, or any combination thereof.

12. 12. The system of claim 11, wherein the AI-accelerator comprises Google-TPU®, Graphcore®, Cerebras®, SambaNova®, or a combination thereof.

13. The system of claim 1 , wherein approximately 1000 VMs run on the GPU.

14. The system of claim 1 , wherein approximately 10,000 VMs run on the GPU.

15. The system of claim 1 , wherein approximately 500 VMs run on the CPU.

16. 10. The system of claim 1, wherein each of the one or more VMs is about 100 to about 500 lines of code.

17. 2. The system of claim 1, wherein each of the one or more VMs is approximately 300 lines of code.

18. The system of claim 1 , wherein the system enables machine programming across a stack.

19. The system of claim 1 , wherein the system further comprises a software stack, a hardware stack, or a hardware-software stack.

20. 20. The system of claim 19, wherein the software stack, the hardware stack, the hardware-software stack, or any combination thereof is differentiable.

21. 20. The system of claim 19, wherein the software stack, the hardware stack, the hardware-software stack, or any combination thereof enables gradient-based learning and / or optimization.

Images