Shadow satisfiability model theory solver system

Abstract

Techniques are described for executing a satisfiability modular theory (SMT) solver in a "shadow" system configuration in which an input query is provided to a primary SMT solver system and additionally to one or more secondary SMT solver systems. The SMT solver system can be used by a cloud provider and used in other computing environments to analyze the effects of configured user account policies that define permissions within the computing environment about user computing resources and associated actions to help ensure security of the computing resources and user data, among other things. The results generated by the primary SMT solver system may be provided to one or more secondary SMT solver systems, where each of the secondary SMT systems may include different system components or different versions of system components to assess correctness of the primary SMT solver system, compare performance metrics, and perform other possible types of analysis.

Description

Background Technology

[0001] Cloud provider networks enable users to access various computing-related resources, such as computing resources, storage resources, and networking resources. When a user or application interacts with a cloud provider network (e.g., using a web-based console, application programming interface (API), or command-line interface (CLI) provided by the cloud provider network), the user or application typically needs to specify security credentials to indicate who the user or application is and whether the user or application has permission to access the requested resource. The cloud provider network then uses these security credentials to authenticate the user or application and authorize the user or application to perform various actions. Access to resources and actions within the cloud provider network can be further managed through policies. A policy is a data object that limits the permissions of a user or resource when associated with it. For example, resource-based policies can be attached to storage resources, compute instances, encryption keys, etc., and can specify who has access to the resource and what actions these identities can perform on the resource. Attached Figure Description

[0002] Various examples according to this disclosure will now be described with reference to the accompanying drawings, in which:

[0003] Figure 1 The diagram illustrates a computing environment based on some examples, which includes an automated inference service from a cloud provider network, comprising a master satisfiability modular theory (SMT) solver system and one or more auxiliary SMT solver systems.

[0004] Figure 2 This is a diagram illustrating formulas expressed in first-order logic based on some examples, which correspond to policy problems involving policies managed by the identity and access management services of a cloud provider's network.

[0005] Figure 3 Additional details are illustrated with some examples of different types of experimental SMT solver systems that can be used to evaluate master SMT solver systems.

[0006] Figure 4 The example illustrates the varying sampling rates that can be used in experimental SMT solver systems based on some examples.

[0007] Figure 5 This is a flowchart illustrating the operation of a method for executing an SMT solver in a “shadow” system configuration, according to some examples, in which input queries are provided to a primary SMT solver system and additionally to one or more auxiliary SMT solver systems for evaluating the accuracy and performance of the primary SMT solver system.

[0008] Figure 6 Examples of provider network environments are provided based on some examples.

[0009] Figure 7 This is a block diagram of an example provider network offering storage services and hardware virtualization services to customers, based on some examples.

[0010] Figure 8 This is a block diagram illustrating an example computer system that can be used in some examples. Detailed Implementation

[0011] This disclosure relates to methods, apparatus, systems, and non-transitory computer-readable storage media for executing a satisfiability modulus theory (SMT) solver in a “shadow” system configuration in which an input query is provided to a primary SMT solver system and additionally to one or more auxiliary SMT solver systems for evaluating the accuracy and performance of the primary SMT solver system. Among other uses, the SMT solver system can be used by cloud providers and in other computing environments to analyze the impact of configured user account policies. These policies define permissions within the computing environment regarding a user’s computing resources and associated actions. The SMT solver system helps ensure the security of computing resources and user data and assists in the development and validation of software and hardware products. As used herein, the SMT solver system may include a modeler (e.g., a software component for transforming a problem provided to the system into a corresponding SMT formula), one or more SMT solvers for determining whether an input SMT formula is satisfiable, and other related components. Based on the example described in this article, the results generated by the main SMT solver system can be provided to one or more auxiliary SMT solver systems, each of which may include different system components or different versions of system components to evaluate the correctness of the main SMT solver system, compare performance metrics, and perform other possible types of analysis.

[0012] SMT solvers are software-based applications that attempt to determine whether a mathematical formula is satisfiable. As noted, the mathematical formulas analyzed by SMT solvers can stem from questions about the configuration of user computing resources in various types of computing environments, policies defining permissions affecting computing resources and other user data, and the state of software or computer systems. Several different types of SMT solvers exist today, and new solvers are constantly being developed, each typically using a unique set of heuristics to solve the problem given to it as input. Because different solvers use different heuristics and other techniques, the time required to solve a given formula can vary significantly between different solvers.

[0013] For distributed systems using SMT solvers to achieve optimal synthesis performance on a wide range of input formulas, many systems employ a "combination" of solvers that process the input formulas in parallel. In some examples, solver combinations can be configured to run in a "race" configuration. In this configuration, the result of the first solver to provide an answer is used, and the others are terminated. This arrangement typically performs well in terms of performance because the most efficient solver is used for each individual query. However, SMT solvers are susceptible to bugs and other defects that can cause solvers to produce incorrect answers to some inputs. Furthermore, newer versions of the same SMT solver may be slower than earlier versions, which can also be undesirable. Therefore, it is desirable to be able to verify the correctness of the solvers and compare the performance of different solvers on a wide range of input queries.

[0014] To address these and other challenges, a system is described in which a "shadow" SMT solver system can be deployed alongside a primary or production SMT solver system. The shadow SMT solver system can be used to run different types of experiments on input queries processed by the primary SMT solver system. For example, an automated inference service of a cloud provider network can receive a request identifying a policy associated with one or more computing resources and attributes expressing the desired conditions of the policy. The service can then invoke the primary SMT solver system based on the policy and attributes. The primary SMT solver system generates a formula based on the policy and attributes and provides the formula to an SMT solver combination to obtain first result data indicating whether the formula is satisfiable or unsatisfiable. The automated inference service can then invoke any number of auxiliary SMT solver systems based on the policy, attributes, and the first result data. These auxiliary SMT solver systems can obtain results from each solver in the solver assembly for comparison with the output from the main SMT solver system. They can include different solver assemblies and different versions of other system components (such as modelers), all of which can be used to compare the accuracy and performance of the main SMT solver system. The ability to easily evaluate the accuracy and performance of an SMT solver system helps improve the security and operation of the user's computing environment, among other benefits, by ensuring accurate and efficient evaluation of its strategies and other system components.

[0015] Figure 1This is a diagram illustrating a computing environment based on some examples, including an automated inference service from a cloud provider network, comprising a Master Satisfiability Modulus Theory (SMT) solver system and one or more auxiliary SMT solver systems. Provider network 100 (or “cloud” provider network) provides users with the ability to use one or more of various types of computing-related resources, such as computing resources (e.g., executing virtual machine (VM) instances and / or containers, executing batch jobs, executing code without provisioning servers), data / storage resources (e.g., object storage, block-level storage, data archive storage, databases and database tables, etc.), network-related resources (e.g., configuring virtual networks (including compute resource groups), content delivery networks (CDNs), domain name services (DNS)), application resources (e.g., databases, application build / deployment services), access policies or roles, identity policies or roles, machine images, routers, and other data processing resources, etc. These and other computing resources may be provided as services, such as hardware virtualization services that can execute computing instances, storage services that can store data objects, etc. Users (or "customers") of provider network 100 may use one or more user accounts associated with a customer account, though these terms may be used interchangeably to some extent depending on the use case. Users may use electronic devices 102 to interact with provider network 100 across one or more intermediate networks 104 (e.g., the Internet) via one or more interfaces, such as through application programming interface (API) calls, via a console implemented as a website or application. An API refers to an interface and / or communication protocol between a client and a server, such that if a client issues a request in a predefined format, the client should receive a response in a specific format or initiate a defined action. In a cloud provider network context, an API provides a gateway to the client to access cloud infrastructure by allowing the client to obtain data from or initiate actions within the cloud provider network, thereby enabling the development of applications that interact with resources and services hosted within the cloud provider network. APIs may also enable different services within the cloud provider network to exchange data with each other. Interfaces may be part of the control plane of provider network 100 or act as the front end of the control plane, which includes "back-end" services that support and implement services that can be provided more directly to the client.

[0016] For example, a cloud provider network (or simply "the cloud") typically refers to a large pool of accessible virtualized computing resources, such as compute, storage, and networking resources, applications, and services. The cloud provides convenient, on-demand network access to a shared pool of configurable computing resources that can be programmatically provisioned and deployed in response to client commands. These resources can be dynamically provisioned and reconfigured to adapt to varying loads. Therefore, cloud computing can be considered both applications delivered as a service via publicly accessible networks (e.g., the internet, cellular networks) and the hardware and software in the cloud provider's data centers that provide these services.

[0017] A cloud provider network can be structured into multiple regions, where a region is a geographical area in which the cloud provider clusters its data centers. Each region comprises multiple (e.g., two or more) Availability Zones (AZs) interconnected via a private high-speed network (e.g., fiber optic communication connection). An AZ (also called a “zone”) provides an isolated fault domain comprising one or more data center facilities, which have separate power supplies, separate networking, and separate cooling relative to data center facilities in another AZ. A data center refers to the physical building or enclosure that houses the servers of the cloud provider network and provides power and cooling to the servers of the cloud provider network. Preferably, AZs within a region are positioned far enough apart that a natural disaster (or other event causing failure) will not simultaneously affect or take down more than one AZ.

[0018] Users can connect to the cloud provider's Availability Zone (AZ) via publicly accessible networks (e.g., the Internet, cellular networks), such as through a Switching Center (TC). A TC is the primary backbone location linking users to the cloud provider's network and can be co-located with other network provider facilities (e.g., Internet Service Providers, telecommunications providers) and securely connected to the AZ (e.g., via VPN or direct connection). Two or more TCs can operate in each region for redundancy. Regions connect to a global network comprising private networking infrastructure (e.g., fiber optic connections controlled by the cloud provider) connecting each region to at least one other region. The cloud provider's network delivers content from access points (or "POPs") located outside but networked with these regions, via edge locations and regional edge caching servers. This partitioning and geographical distribution of computing hardware enables the cloud provider's network to provide users with low-latency resource access globally with high fault tolerance and stability.

[0019] Generally, provider network services and operations can be broadly categorized into two types: control plane operations carried on the logical control plane and data plane operations carried on the logical data plane. The data plane represents the movement of user data through a distributed computing system, while the control plane represents the movement of control signals through the distributed computing system. The control plane typically includes one or more control plane components distributed across one or more control servers and implemented by one or more control servers. Control plane services typically include administrative operations such as system configuration and management (e.g., resource placement, hardware capacity management, diagnostic monitoring, system status information). The data plane includes user resources implemented on the provider network (e.g., compute instances, containers, block storage volumes, databases, file storage). Data plane services typically include non-administrative operations such as transferring user data to and from user resources. Control plane components are typically implemented on a separate set of servers from the data plane servers, and control plane and data plane services can be transmitted over separate / different networks.

[0020] To provide these and other computing resource services, provider network 100 typically relies on virtualization technology. For example, virtualization technology can provide users with the ability to control or use computing resources (e.g., “computing instances,” such as VMs using guest operating systems (O / S) that may or may not operate on top of the underlying host O / S; containers that may or may not operate within VMs; computing instances that can execute on “bare metal” hardware without an underlying hypervisor), where one or more computing resources can be implemented using a single electronic device. Thus, users can directly use computing resources hosted by the provider network (e.g., provided by a hardware virtualization service) to perform various computing tasks. Alternatively or additionally, users can indirectly use computing resources by submitting code to be executed by the provider network (e.g., via an on-demand code execution service), which in turn uses one or more computing resources to execute the code, typically without the user having any control or knowledge of the underlying computing instances involved.

[0021] As described herein, one type of service that a provider network can offer may be referred to as a "managed computing service," which executes code or provides computing resources for its users within a managed configuration. Examples of managed computing services include, for instance, on-demand code execution services, hardware virtualization services, container services, and so on.

[0022] On-Demand Code Execution Service (referred to in various examples as Function Compute Service, Function Service, Cloud Function Service, Function as a Service, or Serverless Compute Service) enables users of Provider Network 100 to execute their code on cloud resources without having to select or manage the underlying hardware resources used to execute the code. For example, a user can use On-Demand Code Execution Service by uploading their code and using one or more APIs to request the service to identify, provision, and manage any resources required to run the code. Therefore, in various examples, "serverless" functionality may include on-demand executable code provided by a user or other entity, such as the Provider Network itself. Serverless functionality may be maintained within the Provider Network through On-Demand Code Execution Service and may be associated with a specific user or account or may typically be accessible to multiple users / accounts. Serverless functionality may be associated with a Uniform Resource Locator (URL), Uniform Resource Identifier (URI), or other reference that can be used to invoke the serverless functionality. Serverless functionality may be executed via computing resources such as virtual machines or containers when triggered or invoked. In some examples, serverless functionality may be invoked via Application Programming Interface (API) calls or specially formatted Hypertext Transfer Protocol (HTTP) request messages. Therefore, users can define the serverless functions that can be executed on demand without requiring them to maintain dedicated infrastructure to execute the serverless functions. Alternatively, resources maintained by the provider network 100 can be used to execute serverless functions on demand. In some examples, these resources may be maintained in a "ready" state (e.g., with a pre-initialized runtime environment configured to execute serverless functions), thereby allowing serverless functions to be executed almost in real time.

[0023] Hardware virtualization services (referred to in various implementations as elastic computing services, virtual machine services, compute cloud services, compute engines, or cloud computing services) enable users of a provider network 100 to provision and manage computing resources, such as virtual machine instances. Virtual machine technology can use a physical server, for example, an equivalent form of running many servers (each referred to as a virtual machine) using a hypervisor, which may run at least on an off-grid card of the server (e.g., a card connected to the physical CPU via PCI or PCIe), and other components of the virtualization host may be available for some virtualization management components. Such an off-grid card of the host may include one or more CPUs that are not available for user instances but are dedicated to instance management tasks, such as virtual machine management (e.g., hypervisor), I / O virtualization of network-attached storage volumes, local migration management tasks, instance health monitoring, etc.). Virtual machines are often referred to as compute instances or simply "instances." As used herein, provisioning a virtual computing instance typically involves: reserving resources (e.g., compute and storage resources) of the underlying physical computing instance for the client (e.g., from a pool of available physical computing instances and other resources); installing or launching the required software (e.g., an operating system); and making the virtual computing instance available to the client to perform client-specified tasks.

[0024] Another type of managed computing service can be a container service that allows users of a cloud provider's network to instantiate and manage containers, such as container orchestration and management services (referred to in various implementations as container services, cloud container services, container engines, or container cloud services). In some examples, a container service can be a Kubernetes-based container orchestration and management service (referred to in various implementations as Kubernetes container services, Azure Kubernetes service, IBM Cloud Kubernetes service, Kubernetes engine, or Kubernetes container engine). As mentioned in this article, containers package code and all its dependencies, so applications (also referred to in various container services as tasks, swarms, or clusters) can run quickly and reliably across various computing environments. A container image is a standalone executable package that includes everything needed to run an application process: code, runtime, system tools, system libraries, and settings. A container image becomes a container at runtime. Therefore, a container is an abstraction of the application layer (meaning each container emulates a different software application process). Although each container runs an isolated process, multiple containers can share a common operating system, for example, by starting within the same virtual machine. In contrast, a virtual machine is an abstraction of the hardware layer (meaning each virtual machine emulates a physical machine that can run software). While multiple virtual machines can run on a single physical machine, each virtual machine typically has its own copy of the operating system, along with the application and its associated files, libraries, and dependencies. Some containers can run on instances running container agents, and some can run on bare metal servers or on off-board cards of the server.

[0025] A Virtual Private Cloud (VPC) (also known as a Virtual Network (VNet), Virtual Private Network, or Virtual Cloud Network in various implementations) is a custom virtual network within another network, such as a cloud provider's network. A VPC can be defined by at least its address space, internal structure (e.g., the compute resources that make up the VPC, security groups), and transit paths, and is logically isolated from other virtual networks in the cloud. A VPC can span all availability zones in a specific region.

[0026] A Virtual Private Cloud (VPC) provides an underlying network layer for cloud services (such as compute cloud or edge cloud) or for customer applications or workloads running on the cloud. A VPC can be dedicated to a specific customer account (or a related set of customer accounts, such as different customer accounts belonging to the same business organization). Customers can contribute resources (such as compute instances) to their VPCs. When creating a VPC, a customer can specify a range of IP addresses for the VPC in the form of a Classless Inter-Domain Routing (CIDR) block. After creating a VPC, a customer can add one or more subnets within each Availability Zone or at an edge location associated with that zone.

[0027] In some examples, Identity and Access Management Service 112 is a service that enables users to securely control access rights to cloud provider network resources (e.g., computing resources associated with various provider network services, such as storage objects associated with storage service 108, databases associated with database services, compute instances associated with hardware virtualization services, etc.). Identity and Access Management Service 112 is broadly used to control who is permitted to authenticate (e.g., log in) to cloud provider network 100 and who is authorized (e.g., has permissions) to use resources provided by cloud provider network 100. Generally, a resource is a concept used to capture a domain of items that can be created, read, modified, or deleted by customers in cloud provider network 100. Examples of resources also include identities (e.g., identity 114, including example users 116A, ..., 116N and roles 118A, ..., 118N) and policies 120 (e.g., including identity-based policy 122, resource-based policy 124, and other possible types of policies). Figure 1 The concept of organization 126 is also illustrated, which may include any number of associated accounts 128, and may also include any number of users and roles.

[0028] When a person initially creates an account through cloud provider network 100, that person may begin to have a single sign-on identity, which has full access to all cloud provider network services and resources associated with that account (e.g., the root user identity 114). For example, the root user identity can be accessed by logging in using the username (e.g., email address) and password used to create the account. Cloud provider network 100 generally recommends that users do not use the root user for most tasks, but instead create supplementary user accounts with limited permissions. Users can grant different permissions to different user accounts for different resources. For example, a user account may be configured to allow some users full access to hardware virtualization services, storage service 108, and other cloud provider network 100 resources. For other users, the user account may allow read-only access to some buckets or permissions to manage some instances, etc.

[0029] In some examples, "subject" refers to a person or application that can issue a request for action or operation on resources of cloud provider network 100 via one or more identities, but sometimes the term "subject" is used interchangeably with "identity." The set of identities 114 associated with account 128 can include any number of users and roles. A cloud provider network request arises when a subject uses an identity (e.g., a user or role) to send a request for action or operation on a resource. The request may include some or all of the following information: the action or operation the subject wants to perform, the resource object on which the action or operation is performed, the person or application that sent the request using an identity (e.g., a user or role), contextual data (e.g., information about IP address, user agent, SSL enabled status, time of day, etc.), and resource data (e.g., data related to the resource being requested, such as a resource identifier or tag name). In some examples, identity and access management service 112 collects the information contained in the request into a request context, which is used to evaluate and authorize the request.

[0030] For some requests to be completed, the Identity and Access Management Service 112 determines whether to authorize (e.g., permit) the requesting subject to complete the request. During authorization, the Identity and Access Management Service 112 uses values ​​included in the request context to examine the policies applicable to the request. The Identity and Access Management Service 112 uses the identified policies to determine whether to allow or deny the request. In some examples, the policies are stored by the Identity and Access Management Service 112 as JavaScript Object Notation (JSON) documents (or using any other data format) and specify permissions for a particular identity. Several types of policies 120 exist that may potentially affect whether a request is authorized, such as, for example, identity-based policies 122, trust policies, and others. For example, to grant a user permission to access a resource within their own account, an identity-based policy can be configured, while a resource-based policy can be used to grant cross-account access to a resource. In some examples, the Identity and Access Management Service 112 examines each policy applicable to the request context. If a single permission policy includes a denied action, the Identity and Access Management Service 112 denies the entire request. In some examples, the Identity and Access Management Service 112 denies requests by default, so that requests are only authorized if each part of the request is permitted by the applicable permission policy.

[0031] Once the request is authenticated and authorized, the Identity and Access Management Service 112 approves the action or operation in the request. The operation is defined by the service and includes actions that can be performed on or relative to a resource, such as viewing, creating, editing, and deleting the resource. For example, the Identity and Access Management Service 112 may support actions such as creating users, deleting users, creating roles, and assuming roles, as well as many other possible actions. The Hardware Virtualization Service may support actions such as starting VM instances and deleting VM instances. To allow a subject to perform an operation, the action is included in a policy applicable to the subject or the affected resource.

[0032] As described in the examples herein, provider network 100 includes an automated inference service 130 to enable the analysis of policies and policy consequences within the cloud provider network. The automated inference service 130 also includes an SMT solver system 160, comprising a solver ensemble including solvers 132 and 142, for enhancing the service's ability to perform automated inference tasks related to policy analysis and other types of problems. As described in more detail herein, the automated inference service 130 also manages one or more experimental SMT solver systems 154 (e.g., including SMT solver systems 156A, ..., SMT solver systems 156N) for checking the correctness, performance, or other characteristics of the SMT solver system 160, which can be used to serve production traffic within the cloud provider network 100.

[0033] like Figure 1As shown, the automated inference service 130 takes one or more policies 134 and one or more attributes 136 (or rules) as input (e.g., "public read / write access to storage resources should not be allowed", "server-side encryption should be enabled for storage resources", etc.). For example, the input received by the automated inference service 130 may be in response to a user request to analyze one or more resources and policies, or may be automatically generated by one or more services of a cloud provider network (e.g., source code analysis service 106, storage service 108, networking service 110, etc.). As described in more detail below, the automated reasoning service 130 broadly obtains one or more policies 134 and attributes 136 at circles "1A" and "1B"; at circle "2", the automated reasoning service 130 uses modeler 138 to model one or more policies 134 and one or more attributes 136 into an SMT formula 140 (shown at circle "3"); and at circle "4", the automated reasoning service 130 provides the SMT formula 140 to solver 132 or a combined solver including multiple solvers including solver 142 to determine the satisfiability of the formula at circle "4". The automated reasoning service 130 uses the results of solver 132 (or one of solvers 142) to inform the user or other systems whether one or more input policies 134 conform to attribute 136, as illustrated by finding result 144 at circle "5". For example, discovery result 144 may alert the user that the policy associated with the user permits public write access to the resource, which may be unintentional, or it may provide discovery result 144 to one or more downstream components that generate alerts, additional analyses, etc. As shown, solver 132 may broadly include a process of encoding the input SMT formula 140 146 into an equivalent satisfiable propositional formula and using SAT solver 148 to determine whether the encoded formula is satisfiable, or, depending on the type of input problem, the solver may broadly use any other process to determine the satisfiability of SMT formula 140. In some examples, SMT solver system 160 uses solver combinations (e.g., solver 132 and solver 142) in parallel and uses the response returned from the first solver to generate an answer as part of the resulting discovery result 144. In this arrangement, solvers that have not yet returned an answer may be terminated, allowing computational resources to be released for subsequent execution.

[0034] As indicated, part of answering questions related to one or more strategies and one or more attributes involves using modeler 138 to transform one or more strategies and attributes into formulas that can be analyzed by the SMT solver or a combination of solvers. Figure 2This is a diagram illustrating formulas expressed in first-order logic based on some examples, corresponding to policy questions involving policies managed by the identity and access management services of a cloud provider's network. As shown, the input to modeler 138 may include one or more policies (e.g., policy 200 including the illustrated policy fragment) and one or more attributes or questions (e.g., attribute 202 indicating the desired conditions of policy 200). In this example, attribute 202 indicates that policy 200 is expected to disallow public write access to any compute resources managed by policy 200. Generally, questions regarding the desired conditions of a policy may include, for example: determining whether public write access to a compute resource is permitted; determining whether unencrypted write access to a compute resource is permitted; determining whether public read access to a compute resource is permitted; or determining whether a Secure Sockets Layer (SSL) request is required to access a compute resource; comparing the permissions of two policies relative to each other, etc.

[0035] In some examples, modeler 138 transforms one or more input strategies and rules into SMT formulas 204 (or SMT queries) involving string theory. Statements may include Boolean combinations (AND, OR, NOT), word equations (e.g., x = const. and x = y, where x is a string variable), and regularization constraints. Figure 2 In the example, formula 204 is a quantifier-free formula expressed in first-order logic.

[0036] As indicated, the automated inference service 130 may use solvers or solver combinations to answer questions involving one or more strategies and one or more associated attributes. Users may also naturally want to verify the correctness of findings generated by the SMT solver system 160 to identify potential vulnerabilities or other misconfigurations associated with the SMT solver system 160, and to evaluate the performance of the SMT solver system compared to other systems, including those with different modeler configurations, different solver combinations, etc. It is further desirable to perform such evaluations in a manner that minimizes the impact on the performance and availability of the SMT solver system 160, which users and other services of the cloud provider network 100 may rely on. Based on the examples described herein, techniques for using one or more auxiliary or “experimental” SMT solver systems in parallel with the primary SMT solver system are described.

[0037] exist Figure 1For example, one or more additional SMT solver systems (e.g., experimental SMT solver system 154 including SMT solver systems 156A, ..., SMT solver system 156N) can be used to reanalyze the result data generated by SMT solver system 160 (e.g., call data 152A, ..., call data 152N) or similar result data from any other source. As shown, call data 152A corresponding to a call to SMT solver system 160 may include request data 162 and corresponding discovery result 144, as well as other possible information. In this example, call data 152A may include, for example, information about the request identifying one or more strategies 134, attributes 136, and any other information about the call to SMT solver system 160 producing discovery result 144. In some examples, the call data may include information reflecting the state of the SMT solver system 160 (including, for example, the version of the SMT solver system or its components), information about the region of the provider network 100 that performs the automated inference service 130, information about resources, tags, or other data accessed by the SMT solver system 160 as part of the call, etc. For example, the state information may be stored as a hash of various data elements or in any other form. In some examples, the call data may optionally include strategies of a different type than those analyzed by the SMT solver system 160, for example, to test the performance of the experimental SMT solver system 154 against a new or modified strategy type.

[0038] In some examples, at circle "6", the automated inference service 130 stores the call data 152A in a queue provided by the publish / subscribe messaging service 150 or in any other type of data repository. For example, each of the call data records may be generated based on the automated inference service 130 calling the SMT system 160 and producing a discovery result. As indicated above, the call to the SMT solver system 160 typically involves the modeler 138 generating a first-order logic formula (or SMT formula 140) based on one or more strategies 134 and one or more attributes 136. The SMT formula 140 is then provided to a first plurality of SMT solvers (e.g., solver 132 and solver 142) to obtain a discovery result 144. The discovery result 144 may then be stored, along with the request data 162 that produced the discovery result 144, in a storage location accessible to one or more experimental SMT solver systems 154. In some examples, at circles "7A", ..., "7N", one or more of the experimental SMT solver systems 154 (e.g., SMT solver systems 156A, ..., SMT solver systems 156N) can independently analyze the same request data as initially analyzed by SMT solver system 160 to determine if the results match. As indicated, these experimental SMT solver systems 154 can be widely used to verify the accuracy of the findings generated by SMT solver system 160, evaluate the performance of SMT solver system 160, etc. For example, the experimental SMT solver systems 154 can collectively or independently generate experimental metrics 158, which indicate the findings generated by the experimental SMT solver systems, identifiers of status information associated with the SMT solver systems that generated the findings, the amount of time elapsed for the SMT solver systems to generate the associated findings, and other possible information. These experimental metrics 158 can be used to identify inconsistencies in results obtained by the experimental SMT solver system compared to the SMT solver system 160, performance comparisons, etc.

[0039] Figure 3 Additional details are illustrated with examples of different types of experimental SMT solver systems that can be used to evaluate master SMT solver systems. Similar to... Figure 1 The automated inference service 130 may include a collection of SMT solver systems 160 (e.g., for production flow) and one or more experimental SMT solver systems 154. Figure 3 The examples show several different types of experimental SMT solver systems, including SMT solver system 156A, SMT solver system 156B, ... and SMT solver system 156N.

[0040] Generally speaking, Figure 3The experimental SMT solver system illustrated illustrates different ways in which it can be used to evaluate a master SMT solver system (e.g., SMT solver system 160). For example, SMT solver system 156A illustrates a configuration in which SMT solver system 156A individually executes each of a plurality of different SMT solvers (e.g., possibly the same set of solvers used by SMT solver system 160) and obtains a result from each solver (e.g., shown as discovery result 300, ..., discovery result 302). As indicated above, in some examples, SMT solver system 160 may execute a solver combination, using the answer obtained from the solver that returned the discovery result earliest compared to the other solvers, and terminating the other solvers in the solver combination. Therefore, the SMT solver system 156A can be used to determine whether any of the other solvers (which may take longer to execute than one or more other solvers in the solver combination) returns findings that differ from those returned by the solvers used by the SMT solver system 160. Differences between the results returned by the solvers of the SMT solver system 156A and the SMT solver system 160 can, for example, indicate a bug or other problem with one or both solvers. Therefore, results from the SMT solver system 156A can be stored as part of experimental metrics 158, and any discrepancies can be used to generate alerts, displayed as part of a metrics dashboard or other interface, or used by any other relevant downstream components. The SMT solver system 156A can also store performance metrics related to execution time, memory usage, or any other metrics related to solver execution for comparison with the SMT solver system 160 or other solvers.

[0041] In some examples, SMT solver system 156B illustrates another example configuration in which one or more different combinations of solvers are used compared to SMT solver system 160. For example, the combination of solvers used in SMT solver system 156B may include one or more additional solvers compared to SMT solver system 160, one or more different versions of the solvers used by SMT solver system 160, one or fewer solvers compared to SMT solver system 160, different configuration settings associated with one or more solvers, or any other differences that might be expected for testing compared to SMT solver system 160. Like SMT solver system 156A, SMT solver system 156B may generate and store experimental metrics 158 that reflect information about the results and performance of SMT solver system 156B against the same call data analyzed by SMT solver system 160 (e.g., shown as discovery result 304). This information can be displayed in various web-based dashboards, provided to downstream components for further analysis, alert generation, or any other purpose.

[0042] In some examples, the SMT solver system 156N exemplifies yet another example configuration in which one or more different versions of the SMT solver system 160 itself are used (e.g., changes to the implementation or configuration of modeler 138, shown as modeler 306 and potentially leading to different SMT formulas 308, or changes to other components of the SMT solver system 160). For example, modeler 138 may include any number of tags and other configurations that determine how modeler 138 translates strategies and properties into corresponding SMT formulas 140. Different versions of the SMT solver system 160 used in the SMT solver system 156N can be used to test different modeler 138 settings, different implementations of modeler 138, or different settings or versions associated with other components of the SMT solver system 160, thereby producing findings 310. Although Figure 3 Three different example types of SMT solver system experiments are shown, but in general, any number of different experimental SMT solver systems 154 can be used as needed to test different solver configurations, modeler versions and configurations, or any other SMT solver system arrangement.

[0043] Figure 4 The diagram illustrates varying sampling rates that can be used in an experimental SMT solver system 154, based on several examples. For instance, see reference... Figure 1The SMT solver system 160 can generate result data for each request processed by the automated inference service 130, and the result data can be stored using a queue 400 provided by the publish / subscribe messaging service 150 or any other storage resource accessible to one or more experimental SMT solver systems 154. Depending on the resource requirements and other considerations of each of the experimental SMT solver systems 154, each of the experimental SMT solver systems 154 can independently analyze all result data generated by the main SMT solver system 160, or only analyze a sampled portion of the result data. Figure 4 As shown, for example, the first SMT solver system 156A can analyze all the result data generated by the SMT solver system 160, while the second SMT solver system 156N can analyze only a sampled portion of the result data generated by the SMT solver system 160. For example, the use of a publish / subscribe messaging service 150 or other storage services enables the experimental SMT solver system 154 to subscribe to queues or other storage resources and invoke the SMT solver for only the sampled result dataset or for result data matching certain conditions of interest.

[0044] While many of the examples described herein are shown in the context of a cloud provider network, the systems described herein involving a primary SMT solver system and one or more experimental SMT solver systems can also be used in other on-premises computing environments, hybrid on-premises and cloud computing environments, etc. For example, the primary SMT solver system 160 may be deployed on a server running in a user's on-premises environment, and one or more experimental SMT solver systems may be further deployed in the same on-premises environment, a separate computing environment, in provider network 100, or other similar arrangements. Furthermore, many of the examples provided involve the use and comparison of SMT solvers. In other examples, automated inference service 130 may use experimental solver systems to verify accuracy and compare performance metrics of the SAT solver or other types of automated inference-related solvers. For example, the SAT solver may be used and tested more directly to analyze issues related to model checking and other problems.

[0045] Figure 5This is a flowchart illustrating operation 500 of a method for executing a satisfiability modulus theory (SMT) solver in a “shadow” system configuration, according to some examples, in which input queries are provided to a primary SMT solver system and additionally to one or more auxiliary SMT solver systems used to evaluate the accuracy and performance of the primary SMT solver system. Some or all of operation 500 (or other processes, variations, and / or combinations thereof described herein) are executed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes collectively on one or more processors. The code is stored, for example, on a computer-readable storage medium in the form of a computer program including instructions executable by one or more processors. The computer-readable storage medium is non-transitory. In some examples, one or more (or all) of operation 500 are executed by an automated inference service of another graph.

[0046] Operation 500 includes receiving, at box 502, a request that identifies a policy associated with one or more computing resources and attributes that express the desired conditions of the policy.

[0047] Operation 500 also includes invoking a first SMT solver system at box 504 based on a strategy and attributes, wherein the first SMT solver system: generates a formula based on the strategy and attributes, and provides the formula to multiple SMT solvers to obtain first result data indicating whether the formula is satisfiable or unsatisfiable.

[0048] Operation 500 also includes invoking the second SMT solver system at box 506 based on the strategy, attributes, and first result data.

[0049] Operation 500 also includes obtaining second result data from the second SMT solver system at box 508.

[0050] Operation 500 also includes providing data at box 510 to another component indicating whether the first result data is different from the second result data.

[0051] In some examples, invoking the first SMT solver system further causes the first SMT solver system to perform the following operations: obtain first result data based on the earliest result received from the SMT solver among the plurality of SMT solvers, and terminate the SMT solver among the plurality of SMT solvers from which the earliest result was not obtained; and wherein the second SMT solver system obtains results from two or more of the plurality of SMT solvers, and wherein the second result data includes results from two or more of the plurality of SMT solvers.

[0052] In some examples, the second SMT solver system differs from the first SMT solver system based on at least one of the following: a different implementation of the modeler for generating formulas based on strategies and attributes, or one or more different configurations of the modeler.

[0053] In some examples, the operation also includes the first SMT solver system storing the first result data in storage resources accessible to both the first and second SMT solver systems; and the second SMT solver system obtaining the first result data from the storage resources.

[0054] In some examples, the operation also includes the first SMT solver system storing multiple result data, including the first result data, in storage resources accessible to both the first and second SMT solver systems; and wherein the second SMT solver system is invoked on a subset of the sampled result data stored in the storage resources.

[0055] In some examples, the second SMT solver system includes a second plurality of SMT solvers that are different from the first plurality of SMT solvers, and the second plurality of SMT solvers differ from the first plurality of SMT solvers based on at least one of the following: the second plurality of SMT solvers includes additional SMT solvers compared to the first plurality of SMT solvers; the second plurality of SMT solvers includes fewer solvers compared to the first plurality of SMT solvers; or the second plurality of SMT solvers includes different versions of the solvers in the first plurality of SMT solvers.

[0056] In some examples, the operation also includes: invoking a third SMT solver system based on the strategy, attributes, and first result data; obtaining third result data from the third SMT solver system; comparing the third result data with the second result data; and causing information indicating the difference between the third result data and the second result data to be displayed.

[0057] In some examples, the operation also includes generating an alert indicating the difference between the first result data and the second result data.

[0058] In some examples, the operation also includes: receiving input identifying multiple requests, each of which identifies a corresponding strategy and a corresponding condition; and invoking a second SMT solver system on each of the multiple requests.

[0059] In some examples, the operation further includes: storing a first index related to a first execution time of the first SMT solver system to obtain first result data by the first SMT solver system; and storing a second index related to a second execution time of the second SMT solver system to obtain second result data by the second SMT solver system.

[0060] In some examples, the resulting data includes information identifying the version of the first SMT solver system and status information associated with the first SMT solver system during invocation of the first SMT solver system.

[0061] Figure 6 An example provider network (or “service provider system”) environment is illustrated according to some examples. Provider network 600 may provide resource virtualization to customers via one or more virtualization services 610, which allow customers to purchase, lease, or otherwise obtain instances 612 of virtualized resources (including, but not limited to, computing and storage resources) implemented on devices within one or more provider networks in one or more data centers. A local Internet Protocol (IP) address 616 may be associated with resource instance 612; the local IP address is the internal network address of resource instance 612 on provider network 600. In some examples, provider network 600 may also provide public IP addresses 614 and / or ranges of public IP addresses (e.g., Internet Protocol version 4 (IPv4) or Internet Protocol version 6 (IPv6) addresses) available to customers from provider 600.

[0062] Typically, provider network 600 may, via virtualization service 610, allow service provider customers (e.g., customers operating one or more customer networks 650A to 650C (or “client networks”) including one or more client devices 652) to dynamically associate at least some public IP addresses 614 assigned to them with specific resource instances 612 assigned to them. Provider network 600 may also allow customers to remap public IP addresses 614 previously mapped to one virtualized computing resource instance 612 assigned to them to another virtualized computing resource instance 612 also assigned to them. Using the virtualized computing resource instances 612 and public IP addresses 614 provided by the service provider, service provider customers, such as operators of customer networks 650A to 650C, may, for example, implement customer-specific applications and present these applications on an intermediate network 640 (such as the Internet). Then, other network entities 620 on the intermediate network 640 can generate traffic destined for a public IP address 614 published by customer networks 650A to 650C; the traffic is routed to the service provider's data center, and at the data center, it is routed via the network substrate to the local IP address 616 of the virtualized computing resource instance 612 currently mapped to the destination public IP address 614. Similarly, response traffic from the virtualized computing resource instance 612 can be routed back to the intermediate network 640 via the network substrate to reach the source entity 620.

[0063] As used herein, a local IP address refers to an internal or “private” network address of a resource instance within, for example, a provider network. Local IP addresses may be within an address block reserved by the Internet Engineering Task Force (IETF) Request for Comments (RFC) 1918 and / or have an address format specified by IETF RFC 4193, and may vary within the provider network. Network traffic originating outside the provider network is not directly routed to a local IP address; instead, traffic uses a public IP address mapped to the local IP address of the resource instance. A provider network may include networking devices or equipment that provide Network Address Translation (NAT) or similar functionality to perform mappings from public IP addresses to local IP addresses and from local IP addresses to public IP addresses.

[0064] A public IP address is a variable network address on the Internet assigned to a resource instance by a service provider or customer. For example, traffic routed to a public IP address via 1:1 NAT translation is used to forward the traffic to the appropriate local IP address of the resource instance.

[0065] Some public IP addresses may be assigned to specific resource instances by the provider's network infrastructure; these public IP addresses may be referred to as standard public IP addresses, or simply standard IP addresses. In some examples, the mapping of standard IP addresses to the local IP addresses of resource instances is the default startup configuration for all resource instance types.

[0066] At least some public IP addresses can be assigned to or obtained by customers of Provider Network 600; customers can then assign their assigned public IP addresses to specific resource instances assigned to them. These public IP addresses may be referred to as customer public IP addresses, or simply customer IP addresses. Instead of being assigned to resource instances by Provider Network 600 as in the case of standard IP addresses, customer IP addresses can be assigned to resource instances by the customer, for example, via an API provided by the service provider. Unlike standard IP addresses, customer IP addresses are assigned to customer accounts and can be remapped to other resource instances by the relevant customer as needed. Customer IP addresses are associated with customer accounts, not specific resource instances, and the customer controls the IP address until the customer chooses to release it. Unlike regular static IP addresses, customer IP addresses allow customers to mask resource instance or availability zone failures by remapping their public IP addresses to any resource instance associated with their customer account. For example, customer IP addresses enable customers to resolve resource instance or software issues by remapping their customer IP addresses to alternative resource instances.

[0067] Figure 7This is a block diagram of an example provider network environment that provides storage services and hardware virtualization services to customers, based on some examples. Hardware virtualization service 720 provides customers with multiple computing resources 724 (e.g., computing instances 725 such as VMs). Computing resources 724 may be provided as a service to customers of provider network 700 (e.g., customers implementing customer network 750). Each computing resource 724 may have one or more local IP addresses. Provider network 700 may be configured to route packets from the local IP addresses of computing resources 724 to public internet destinations, and to route packets from public internet sources to the local IP addresses of computing resources 724.

[0068] Provider network 700 may provide a client network 750, for example, coupled to intermediate network 740 via local network 756, with the ability to implement virtual computing systems 792 via hardware virtualization service 720 coupled to intermediate network 740 and provider network 700. In some examples, hardware virtualization service 720 may provide one or more APIs 702, such as web service interfaces, through which client network 750 may access the functionality provided by hardware virtualization service 720, for example, via console 794 of client device 790 (e.g., web-based applications, standalone applications, mobile applications, etc.). In some examples, at provider network 700, each virtual computing system 792 at client network 750 may correspond to computing resources 724 that are leased, rented, or otherwise provided to client network 750.

[0069] From an instance of virtual computing system 792 and / or another client device 790 (e.g., via console 794), a client can access the functionality of storage service 710, for example, via one or more APIs 702, to access data from and store data in storage resources 718A to 718N of a virtual data repository 716 (e.g., folders or "buckets," virtualized volumes, databases, etc.) provided by provider network 700. In some examples, a virtualized data storage gateway (not shown) may be located at client network 750, which may locally cache at least some data (e.g., frequently accessed data or critical data) and communicate with storage service 710 via one or more communication channels to upload new or modified data from the local cache, thereby maintaining the master data repository (virtualized data repository 716). In some examples, a user can mount and access volumes of virtualized data repository 716 via storage service 710, which acts as a storage virtualization service, and these volumes may appear to the user as local (virtualized) storage 798.

[0070] Although Figure 7Although not shown, the virtualization service can also be accessed from resource instances within provider network 700 via API 702. For example, a customer, equipment service provider, or other entity can access the virtualization service from within a corresponding virtual network on provider network 700 via API 702 to request the allocation of one or more resource instances within said virtual network or another virtual network.

[0071] In some examples, systems implementing some or all of the techniques described herein may include general-purpose computer systems, such as Figure 8 The illustrated computer system 800 (also referred to as a computing device or electronic device), this general-purpose computer system includes or is configured to access one or more computer-accessible media. In the illustrated example, computer system 800 includes one or more processors 810 coupled to system memory 820 via input / output (I / O) interface 830. Computer system 800 also includes a network interface 840 coupled to I / O interface 830. Although Figure 8 The computer system 800 is shown as a single computing device, but in various examples, the computer system 800 may include a single computing device or any number of computing devices configured to work together as a single computer system 800.

[0072] In various examples, computer system 800 may be a single-processor system including one processor 810 or a multiprocessor system including several processors 810 (e.g., two, four, eight, or another suitable number). Processor 810 may be any suitable processor capable of executing instructions. For example, in various examples, processor 810 may be a general-purpose or embedded processor implementing any of a variety of instruction set architectures (ISAs), such as x86, ARM, PowerPC, SPARC, or the MIPS ISA or any other suitable ISA. In a multiprocessor system, each of the processors 810 may, but does not necessarily, implement the same ISA.

[0073] System memory 820 may store instructions and data accessible to processor 810. In various examples, system memory 820 may be implemented using any suitable memory technology, such as random access memory (RAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory. In the illustrated example, program instructions and data that implement one or more desired functions (such as the methods, techniques, and data described above) are shown stored in system memory 820 as automated inference service code 825 (e.g., executable to fully or partially implement automated inference service 130) and data 826.

[0074] In some examples, I / O interface 830 may be configured to coordinate I / O traffic between processor 810, system memory 820, and any peripheral devices within the device, including network interface 840 and / or other peripheral interfaces (not shown). In some examples, I / O interface 830 may perform any necessary protocol, timing, or other data transformations to convert data signals from one component (e.g., system memory 820) into a format suitable for use by another component (e.g., processor 810). In some examples, for instance, I / O interface 830 may include devices that support attachment via various types of peripheral buses, such as the Peripheral Component Interconnect (PCI) bus standard or variants of the Universal Serial Bus (USB) standard. In some examples, for instance, the functionality of I / O interface 830 may be split into two or more separate components, such as a northbridge and a southbridge. Additionally, in some examples, some or all of the functionality of I / O interface 830 (such as an interface to system memory 820) may be directly incorporated into processor 810.

[0075] For example, network interface 840 may be configured to allow computer system 800 to communicate with other devices 860 attached to one or more networks 850 (such as, Figure 1 The network interface 840 may exchange data with other computer systems or devices as illustrated. In various examples, for instance, the network interface 840 may support communication via any suitable wired or wireless general-purpose data network, such as various types of Ethernet networks. Additionally, the network interface 840 may support communication via telecommunications / telephone networks (such as analog voice networks or digital fiber optic communication networks), via storage area networks (SANs) (such as Fibre Channel SANs), and / or via any other suitable type of network and / or protocol.

[0076] In some examples, computer system 800 includes one or more offload cards 870A or 870B (including one or more processors 875 and possibly one or more network interfaces 840), which are connected using I / O interfaces 830 (e.g., a bus implementing a version of the Peripheral Component Interconnect Fast (PCI-E) standard or another interconnect such as Fast Path Interconnect (QPI) or Hyper Path Interconnect (UPI). For example, in some examples, computer system 800 may act as a host electronic device hosting computing resources such as compute instances (e.g., operating as part of a hardware virtualization service), and one or more offload cards 870A or 870B act as a virtualization manager that manages the compute instances running on the host electronic device. As an example, in some examples, offload card 870A or 870B may perform compute instance management operations such as pausing and / or unpausing compute instances, starting and / or terminating compute instances, performing memory transfer / copy operations, etc. In some examples, these management operations may be performed by the offload card 870A or 870B in cooperation with a hypervisor executed by other processors 810A to 810N of the computer system 800 (e.g., upon request from the hypervisor). However, in some examples, the virtualization manager implemented by the offload card 870A or 870B may adapt to requests from other entities (e.g., from the computing instance itself) and may not cooperate with (or serve) any single hypervisor.

[0077] In some examples, system memory 820 may be one example of a computer-accessible medium configured to store program instructions and data as described above. However, in other examples, program instructions and / or data may be received, transmitted, or stored on different types of computer-accessible media. Generally, computer-accessible media may include any non-transitory storage medium or memory medium, such as magnetic or optical media, for example, a disk or DVD / CD coupled to computer system 800 via I / O interface 830. Non-transitory computer-accessible storage media may also include any volatile or non-volatile medium that may be included as system memory 820 or another type of memory in some examples of computer system 800, such as RAM (e.g., SDRAM, Double Data Rate (DDR) SDRAM, SRAM, etc.), read-only memory (ROM), etc. Furthermore, computer-accessible media may include transmission media or signals transmitted via communication media (such as networks and / or wireless links), such as electrical signals, electromagnetic signals, or digital signals, such as those implemented via network interface 840.

[0078] The various examples discussed or presented herein can be implemented in a wide variety of operating environments, in some cases of which may include one or more user computers, computing devices, or processing devices that can be used to operate any of a number of applications. User devices or client devices may include any of a number of general-purpose personal computers, such as desktop or laptop computers running standard operating systems, and cellular, wireless, and handheld devices running mobile software and capable of supporting a number of networking and messaging protocols. Such systems may also include a number of workstations running a variety of commercially available operating systems and any of other known applications for purposes such as development and database management. These devices may also include other electronic devices, such as virtual terminals, thin clients, gaming systems, and / or other devices capable of communicating via a network.

[0079] Most examples use at least one network familiar to those skilled in the art to support communication using any of a wide range of widely available protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), File Transfer Protocol (FTP), Universal Plug and Play (UPnP), Network File System (NFS), Public Internet File System (CIFS), Extensible Messaging and Attendance Protocol (XMPP), AppleTalk, etc. The network may include, for example, a Local Area Network (LAN), a Wide Area Network (WAN), a Virtual Private Network (VPN), the Internet, an intranet, an extranet, the Public Switched Telephone Network (PSTN), an infrared network, a wireless network, and any combination thereof.

[0080] In examples using a web server, the web server can run any of a variety of server or middleware applications, including HTTP servers, File Transfer Protocol (FTP) servers, Common Gateway Interface (CGI) servers, data servers, Java servers, business application servers, etc. The server may also be able to execute programs or scripts in response to requests from user devices, such as by executing one or more web applications that can be implemented as one or more scripts or programs written in any programming language (such as Java®, C, C#, or C++) or any scripting language (such as Perl, Python, PHP, or TCL) and combinations thereof. The server may also include a database server, including but not limited to commercially available database servers from Oracle®, Microsoft®, Sybase®, IBM®, etc. The database server can be relational or non-relational (e.g., "NoSQL"), distributed or non-distributed, etc.

[0081] The environment disclosed herein may include various data repositories and other storage and storage media as discussed above. These may reside in various locations, such as on (and / or in) storage media local to one or more computers, or on storage media of any or all computers remote from a network. In a particular set of examples, information may reside in a storage area network (SAN) familiar to those skilled in the art. Similarly, any necessary files for performing functions belonging to a computer, server, or other network device may be stored locally and / or remotely, depending on the circumstances. Where the system includes computerized devices, each such device may include hardware elements electrically coupled via a bus, including, for example, at least one central processing unit (CPU), at least one input device (e.g., mouse, keyboard, controller, touchscreen, or keypad), and / or at least one output device (e.g., display device, printer, or speaker). Such a system may also include one or more storage devices, such as hard disk drives, optical storage devices, and solid-state storage devices such as random access memory (RAM) or read-only memory (ROM), as well as removable media devices, memory cards, flash memory cards, etc.

[0082] Such devices may also include computer-readable storage medium readers, communication devices (e.g., modems, network interface cards (wireless or wired), infrared communication devices, etc.), and working memory as described above. A computer-readable storage medium reader may be connected to or configured to receive a computer-readable storage medium, which represents a remote, local, fixed, and / or removable storage device and storage medium for temporarily and / or more permanently accommodating, storing, transmitting, and retrieving computer-readable information. The systems and various devices will also typically include numerous software applications, modules, services, or other elements, including operating systems and applications such as client applications or web browsers, residing within at least one working memory device. It should be understood that alternative examples may have many variations different from those described above. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software (including portable software such as applets), or both. Furthermore, connectivity to other computing devices, such as network input / output devices, may be employed.

[0083] Storage media and computer-readable media used to contain code or code portions may include any suitable media known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile media, removable and non-removable media implemented in any way or technology to store and / or transmit information (such as computer-readable instructions, data structures, program modules or other data), including RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc-read-only memory (CD-ROM), digital universal disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by system devices. Based on this disclosure and the teachings provided herein, those skilled in the art will appreciate other ways and / or methods for implementing the various examples.

[0084] In the foregoing description, various examples have been described. Specific configurations and details have been elaborated for illustrative purposes to provide a thorough understanding of the examples. However, it will also be apparent to those skilled in the art that the examples can be practiced without these specific details. Furthermore, well-known features may be omitted or simplified to avoid obscuring the described examples.

[0085] In this document, parenthesized text and boxes with dashed borders (e.g., large dashes, small dashes, dot dashes, and dots) are used to illustrate optional aspects for adding additional features to some examples. However, this notation should not be interpreted as implying that these are the only options or optional operations, and / or that in some examples, boxes with solid borders are not optional.

[0086] Reference numerals with suffix letters (e.g., 718A to 718N) can be used to indicate that one or more instances of the mentioned entity may exist in various examples, and when multiple instances exist, each instance need not be identical, but may share some general characteristics or function in a common form. Furthermore, unless explicitly indicated to the contrary, the use of a particular suffix does not imply the existence of a specific number of entities. Thus, in various examples, two entities using the same or different suffix letters may have or not have the same number of instances.

[0087] References to "an example," "example," etc., indicate that the described example may include a specific feature, structure, or characteristic, but each example may not necessarily include the specific feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same example. Additionally, when a specific feature, structure, or characteristic is described in conjunction with an example, it should be assumed that, whether explicitly described or not, implementing such a feature, structure, or characteristic in conjunction with other examples is within the knowledge of those skilled in the art.

[0088] Furthermore, in the various examples described above, unless otherwise specifically indicated, the disjunctive linguistic intent of phrases such as “at least one of A, B, or C” is understood to mean A, B, or C, or any combination thereof (e.g., A, B, and / or C). Similarly, the linguistic intent of phrases such as “at least one or more of A, B, and C” (or “one or more of A, B, and C”) is understood to mean A, B, or C, or any combination thereof (e.g., A, B, and / or C). Therefore, disjunctive language is neither intended nor should be understood to imply that a given example requires the existence of at least one of A, at least one of B, and at least one of C.

[0089] As used herein, the term "based on" (or similar) is an open-ended term used to describe one or more factors that influence a determination or other action. It should be understood that this term does not exclude additional factors that may influence a determination or action. For example, a determination may be based solely on the listed factors or on said factors and one or more additional factors. Therefore, if action A is "based on" B, it should be understood that B is a factor influencing action A, but this does not preclude the action from also being based on one or more other factors, such as factor C. However, in some cases, action A may be entirely based on B.

[0090] Unless otherwise expressly stated, articles such as “a / an” should generally be interpreted as including one or more of the described items. Therefore, phrases such as “a device configured to…” or “computing device” are intended to include one or more of the described devices. Such one or more described devices may be collectively configured to perform the described operations. For example, “a processor configured to perform operations A, B, and C” may include a first processor configured to perform operation A working in conjunction with a second processor configured to perform operations B and C.

[0091] Furthermore, the words “may” or “may” are used in a permissive sense (i.e., implying a possibility) rather than a mandatory sense (i.e., implying a requirement). The words “include,” “including,” and “includes” are used to indicate an open relationship and therefore imply, including but not limited to. Similarly, the words “have,” “having,” and “has” also indicate an open relationship and therefore imply, having but not limited to. Terms such as “first,” “second,” “third,” etc., as used herein, serve as markers for the nouns that follow them and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless otherwise explicitly indicated. Similarly, the values ​​of such numerical markers are generally not used to indicate the required quantity of a particular noun in a claim set forth herein, and therefore, the element “fifth” generally does not imply the presence of four other elements unless those elements are explicitly included in the claim or their presence is otherwise sufficiently clear.

[0092] At least some embodiments of the disclosed technology may be described in light of the following terms:

[0093] Clause 1: A computer-implemented method comprising: receiving a request by an automated inference service of a cloud provider network, the request identifying: a policy managed by an identity and access management service of the cloud provider network, and attributes expressing desired conditions of the policy; invoking a first SMT solver system based on the policy and the attributes, wherein the first SMT solver system includes a modeler and a plurality of SMT solvers, wherein invoking the first SMT solver system causes the first SMT solver system to: generate a first-order logic formula by the modeler based on the policy and the attributes, and provide the first-order logic formula to the plurality of SMT solvers to obtain first result data indicating whether the first-order logic formula is satisfiable or unsatisfiable; invoking a second SMT solver system based on the policy, the attributes, and the first result data; obtaining second result data from the second SMT solver system; and displaying data indicating whether the first result data is different from the second result data.

[0094] Clause 2: A computer-implemented method as described in Clause 1, wherein invoking the first SMT solver system further causes the first SMT solver system to perform the following operations: obtaining first result data based on the earliest result received from one of the plurality of SMT solvers, and terminating the SMT solvers from which the earliest result was not obtained; and wherein the second SMT solver system obtains results from two or more of the plurality of SMT solvers, and wherein the second result data includes the results from two or more of the plurality of SMT solvers.

[0095] Clause 3: The computer-implemented method as described in Clause 1, wherein the second SMT solver system differs from the first SMT solver system based on at least one of the following: a different implementation of the modeler, or one or more different configurations of the modeler.

[0096] Clause 4: A computer-implemented method comprising: receiving a request identifying a strategy associated with one or more computing resources and attributes expressing desired conditions of the strategy; invoking a first SMT solver system based on the strategy and the attributes, wherein the first SMT solver system: generates a formula based on the strategy and the attributes, and provides the formula to a plurality of SMT solvers to obtain first result data indicating whether the formula is satisfiable or unsatisfiable; invoking a second SMT solver system based on the strategy, the attributes, and the first result data; obtaining second result data from the second SMT solver system; and providing data to another component indicating whether the first result data is different from the second result data.

[0097] Clause 5: A computer-implemented method as described in Clause 4, wherein invoking the first SMT solver system further causes the first SMT solver system to: obtain first result data based on the earliest result received from the SMT solvers among the plurality of SMT solvers, and terminate the SMT solvers from which the earliest result was not obtained; and wherein the second SMT solver system obtains results from two or more of the plurality of SMT solvers, and wherein the second result data includes results from two or more of the plurality of SMT solvers.

[0098] Clause 6: The computer-implemented method as described in Clause 4, wherein the second SMT solver system differs from the first SMT solver system based on at least one of the following: a different implementation of the modeler for generating the formula based on the strategy and the properties, or one or more different configurations of the modeler.

[0099] Clause 7: The computer-implemented method as described in Clause 4 further includes: storing the first result data by the first SMT solver system in a storage resource accessible to both the first SMT solver system and the second SMT solver system; and obtaining the first result data from the storage resource by the second SMT solver system.

[0100] Clause 8: The computer-implemented method as described in Clause 4 further comprises: storing a plurality of result data, including the first result data, in storage resources accessible to the first SMT solver system and the second SMT solver system by the first SMT solver system; and wherein the second SMT solver system is invoked on a subset of samples of the plurality of result data stored in the storage resources.

[0101] Clause 9: A computer-implemented method as described in Clause 4, wherein the second SMT solver system includes a second plurality of SMT solvers that are different from the first plurality of SMT solvers, and wherein the second plurality of SMT solvers differs from the first plurality of SMT solvers based on at least one of the following: the second plurality of SMT solvers includes additional SMT solvers compared to the first plurality of SMT solvers; the second plurality of SMT solvers includes fewer solvers compared to the first plurality of SMT solvers; or the second plurality of SMT solvers includes different versions of the solvers in the first plurality of SMT solvers.

[0102] Clause 10: The computer-implemented method as described in Clause 4 further includes: invoking a third SMT solver system based on the strategy, the attribute, and the first result data; obtaining third result data from the third SMT solver system; comparing the third result data with the second result data; and causing information indicating the difference between the third result data and the second result data to be displayed.

[0103] Clause 11: The computer-implemented method as described in Clause 4 further includes generating an alarm indicating a difference between the first result data and the second result data.

[0104] Clause 12: The computer-implemented method as described in Clause 4 further includes: receiving input identifying a plurality of requests, each of the plurality of requests identifying a corresponding strategy and a corresponding condition; and invoking the second SMT solver system on each of the plurality of requests.

[0105] Clause 13: The computer-implemented method as described in Clause 4 further includes: storing a first index related to a first execution time of the first SMT solver system to obtain the first result data; and storing a second index related to a second execution time of the second SMT solver system to obtain the second result data.

[0106] Clause 14: The computer-implemented method as described in Clause 4, wherein the result data includes information identifying the version of the first SMT solver system and status information associated with the first SMT solver system during invocation of the first SMT solver system.

[0107] Clause 15: A system comprising: a first or more electronic devices configured to implement an automated inference service in a multi-tenant provider network, wherein the automated inference service includes instructions that, when executed, cause the automated inference service to: receive a request identifying: a policy managed by an identity and access management service of the multi-tenant provider network, and attributes expressing desired conditions of the policy; invoke a first SMT solver system based on the policy and the attributes, wherein the first SMT solver system includes a modeler and a plurality of SMT solvers to obtain first result data; and invoke a second SMT solver system based on the policy, the attributes, and the first result data. A solver system; obtaining second result data from a second SMT solver system; and causing data to be displayed indicating whether the first result data is different from the second result data; and a second or more electronic devices for implementing the first SMT solver system in the multi-tenant provider network, wherein the first SMT solver system includes instructions that, when executed, cause the first SMT solver system to: generate a first-order logic formula by the modeler based on the strategy and the attribute, and provide the first-order logic formula to the plurality of SMT solvers to obtain first result data indicating whether the first-order logic formula is satisfiable or unsatisfiable.

[0108] Clause 16: The system as described in Clause 15, wherein invoking the first SMT solver system further causes the first SMT solver system to perform the following operations: obtain the first result data based on the earliest result received from the SMT solvers among the plurality of SMT solvers, and terminate the SMT solvers from which the earliest result was not obtained; and wherein the second SMT solver system obtains results from two or more of the plurality of SMT solvers, and wherein the second result data includes the results from two or more of the plurality of SMT solvers.

[0109] Clause 17: A system as described in Clause 15, wherein the second SMT solver system differs from the first SMT solver system based on at least one of the following: a different implementation of the modeler, or one or more different configurations of the modeler.

[0110] Clause 18: The system as described in Clause 15, wherein the instructions, when executed by the automated inference service, further cause the automated inference service to: store the first result data in a queue provided by a service of a cloud provider network by the first SMT solver system; and obtain the first result data from the queue by the second SMT solver system.

[0111] Clause 19: The system as described in Clause 15, wherein the instructions, when executed by the automated inference service, further cause the automated inference service to: store a plurality of result data, including the first result data, in storage resources provided by a cloud provider network by the first SMT solver system; and wherein the second SMT solver system is invoked on a subset of samples of the plurality of result data.

[0112] Clause 20: The system as described in Clause 15, wherein the second SMT solver system includes a second plurality of SMT solvers that are different from the first plurality of SMT solvers.

[0113] The specification and drawings should therefore be considered illustrative rather than restrictive. However, it will be apparent that various modifications and changes may be made therein without departing from the broader scope of this disclosure as set forth in the claims.

Claims

1. A computer-implemented method, comprising: Receive a request, the request identifying a strategy associated with one or more computing resources and attributes expressing the desired conditions of the strategy; The first SMT solver system is invoked based on the strategy and the attributes, wherein the first SMT solver system is: Formulas are generated based on the strategy and the attributes, and The formula is provided to multiple SMT solvers to obtain first result data indicating whether the formula is satisfiable or unsatisfiable; The second SMT solver system is invoked based on the strategy, the attributes, and the first result data; The second result data is obtained from the second SMT solver system; as well as Provide another component with data indicating whether the first result data is different from the second result data.

2. The computer-implemented method of claim 1, wherein invoking the first SMT solver system further causes the first SMT solver system to perform the following operations: obtaining the first result data based on the earliest result received from the SMT solver among the plurality of SMT solvers, and terminating the SMT solver among the plurality of SMT solvers from which the earliest result was not obtained; and The second SMT solver system obtains results from two or more of a plurality of SMT solvers, and the second result data includes results from two or more of the plurality of SMT solvers.

3. The computer-implemented method of claim 1, wherein the second SMT solver system differs from the first SMT solver system based on at least one of the following: a different implementation of the modeler for generating the formula based on the strategy and the properties, or one or more different configurations of the modeler.

4. The computer-implemented method as described in claim 1, further comprising: The first result data is stored by the first SMT solver system in storage resources accessible to both the first SMT solver system and the second SMT solver system. as well as The second SMT solver system obtains the first result data from the storage resource.

5. The computer-implemented method as described in claim 1, further comprising: The first SMT solver system stores multiple result data, including the first result data, in storage resources accessible to both the first SMT solver system and the second SMT solver system. and The second SMT solver system is invoked on a subset of the samples of the plurality of result data stored in the storage resource.

6. The computer-implemented method of claim 1, wherein the second SMT solver system comprises a second plurality of SMT solvers different from the first plurality of SMT solvers, and wherein the second plurality of SMT solvers differs from the first plurality of SMT solvers based on at least one of the following: the second plurality of SMT solvers includes additional SMT solvers compared to the first plurality of SMT solvers; the second plurality of SMT solvers includes fewer solvers compared to the first plurality of SMT solvers; or the second plurality of SMT solvers includes different versions of the solvers in the first plurality of SMT solvers.

7. The computer-implemented method as described in claim 1, further comprising: The third SMT solver system is invoked based on the strategy, the attributes, and the first result data; The third result data is obtained from the third SMT solver system; Compare the third result data with the second result data; as well as This enables the display of information indicating the difference between the third result data and the second result data.

8. The computer-implemented method of claim 1, further comprising generating an alarm indicating a difference between the first result data and the second result data.

9. The computer-implemented method as described in claim 1, further comprising: Receive input identifying multiple requests, each of which identifies a corresponding policy and a corresponding condition; as well as The second SMT solver system is invoked on each of the plurality of requests.

10. The computer-implemented method of claim 1, further comprising: The first result data is obtained by storing a first index related to the first execution time of the first SMT solver system; as well as The second result data is obtained by storing a second index related to the second execution time of the second SMT solver system.

11. The computer-implemented method of claim 1, wherein the result data includes information identifying the version of the first SMT solver system and status information associated with the first SMT solver system during invocation of the first SMT solver system.

12. The computer-implemented method of any one of claims 1 to 11, wherein the request identifying the policy and the attribute is received by an automated inference service of a cloud provider network; and wherein the policy is managed by an identity and access management service of the cloud provider network.

13. The computer-implemented method of any one of claims 1 to 11, wherein the first SMT solver system includes a modeler and a plurality of SMT solvers; wherein the first SMT solver system is invoked such that the modeler generates the formula based on the strategy and the attribute; and wherein the formula is a first-order logic formula.

14. The computer-implemented method of any one of claims 1 to 11, wherein providing data to another component indicating whether the first result data is different from the second result data includes causing the display of data indicating whether the first result data is different from the second result data.

15. A system comprising: One or more electronic devices for implementing one or more services in a multi-tenant provider network, wherein the one or more services include instructions that, when executed, cause the one or more services to perform the method as described in any one of claims 1 to 14.

Images