Arrangement of grid

Abstract

The invention discloses orchestration of grids. Methods, apparatus, systems, and articles of manufacture are disclosed for orchestrating execution of services in an orchestrator grid of an edge cloud computing environment. An example orchestrator apparatus includes an interface to receive a service to execute and to monitor execution of the service. An example apparatus includes an orchestration delegate to select a resource and orchestrate a deployment of a service to perform using the selected resource. An example apparatus includes a delegate orchestration manager to manage an orchestration delegate based on information from an interface and an orchestration delegate related to execution of a service.

Description

[0001] This application is a divisional application of application No. 202110864865.2, filed on July 29, 2021, entitled "Organization of Grids". Technical Field

[0002] This disclosure generally relates to edge computing environments, and more specifically to the orchestration of grids in edge computing environments. Background Technology

[0003] Edge computing resources can be used to perform services and other applications on behalf of various tenants. Some cloud systems may include multiple edges. Each edge can be managed separately with different carriers, different policies, different architectures, etc. Therefore, coordinating activities across multiple edges of the network infrastructure is difficult (if not impossible). Attached Figure Description

[0004] Figure 1 Describe the example usage environments, including example cloud environments, example edge environments, and example endpoint environments.

[0005] Figure 2 This is an overview of a sample edge cloud configuration for edge computing.

[0006] Figure 3 The diagram illustrates the operational layer between endpoints, edge cloud, and cloud computing environments.

[0007] Figure 4 The diagram illustrates a block diagram of an example environment for networking and services in an edge computing system.

[0008] Figure 5 The diagram illustrates the deployment of a virtual edge configuration in an edge computing system operating across multiple edge nodes and multiple tenants.

[0009] Figure 6 The diagram illustrates various computing arrangements for deploying containers in an edge computing system.

[0010] Figure 7 These are example computing and communication use cases involving mobile access to applications in edge computing systems.

[0011] Figure 8A It can be deployed in Figures 1-5 and / or Figure 7 The diagram illustrates a block diagram of an example implementation of an example computing node in one of the edge computing systems.

[0012] Figure 8B It can be deployed in Figures 1-5 and / or Figure 7Another block diagram illustrating an example implementation of an example computing node in one of the edge computing systems shown in the figure.

[0013] Figure 9 An example edge architecture is shown.

[0014] Figure 10 It is used with example edge architectures (such as Figure 9 A block diagram of an example orchestration system used in conjunction with an edge architecture.

[0015] Figure 11 The diagram illustrates the use of Figure 10 This is an example network environment or infrastructure implemented using an example orchestration system.

[0016] Figure 12 The diagram illustrates another example network architecture used for delegation to implement orchestration.

[0017] Figure 13 This is a block diagram of an example fog network, which includes fog management domains implemented in the cloud and / or at the edge.

[0018] Figure 14 This is an example of a network infrastructure in which multiple edge / cloud resources accommodate multiple fog management domains to coordinate resource allocation, configuration, and / or other activities among multiple fog nodes.

[0019] Figure 15 This illustrates a sample data flow between the orchestrator and the verification server, used to configure orchestration delegation(s) and facilitate the execution of services and the allocation of additional resources(s) for executing services.

[0020] Figures 16-17 This indicates that it can be executed to achieve [the desired result]. Figures 10-15 A flowchart of example machine-readable instructions for an example network computing infrastructure.

[0021] Figure 18 It is constructed for execution Figures 16-17 Instructions to achieve Figures 10-15 A block diagram of an example processor platform for example network computing infrastructure.

[0022] These figures are not drawn to scale. Generally, throughout the figures and accompanying written description, the same reference numerals will be used to refer to the same or similar parts. Unless otherwise indicated, connection references (e.g., attached, coupled, connected, and combined) should be interpreted broadly and may include intermediate members between sets of elements and relative movement between elements. Thus, a connection reference does not necessarily imply that two elements are directly connected and in a fixed relationship with each other. Detailed Implementation

[0023] When identifying multiple elements or components, this document uses descriptors such as “first,” “second,” “third,” etc. Unless otherwise specified or understood based on their context of use, such descriptors are not intended to assign any meaning of priority, physical order, or arrangement in a list, or chronological order, but are merely labels used to separately refer to multiple elements or components for ease of understanding of the disclosed examples. In some examples, the descriptor “first” may be used to refer to an element in a detailed description, while different descriptors such as “second” or “third” may be used in the claims to refer to the same element. In such cases, it should be understood that such descriptors are used only for ease of reference to multiple elements or components. As used herein, “approximately” and “about” refer to dimensions that may not be precise due to manufacturing tolerances and / or other real-world defects. As used herein, “substantially real-time” means that, recognizing real-world delays in computation time, transmission, etc., occur in a near-instantaneous manner. Thus, unless otherwise specified, “substantially real-time” means real-time + / - 1 second.

[0024] In general, edge computing refers to the shift of computing and storage resources closer to endpoint devices (e.g., consumer computing devices, user rigs, etc.) to optimize total cost of ownership, reduce application latency, improve service capabilities, and enhance compliance with security and / or data privacy requirements. In some scenarios, edge computing provides cloud-like distributed services that orchestrate and manage various types of storage and computing resources for applications. As a result, some implementations of edge computing are referred to as “edge cloud” or “fog” because powerful computing resources previously available only in large, remote data centers are moved closer to the endpoints, making them available to consumers at the “edge” of the network.

[0025] Edge computing use cases, also known as "mobile edge computing," have been developed that utilize mobile network configurations for integration with multi-access edge computing (MEC). MEC is designed to allow application developers and content providers to access the computing power and information technology (IT) service environment at the edge of the network with dynamic mobile network configurations. The European Telecommunications Standards Institute (ETSI) Industry Specification Group (ISG) has developed a limited set of standards attempting to define common interfaces for operating MEC systems, platforms, hosts, services, and applications.

[0026] Edge computing, satellite edge computing (e.g., edge nodes connected to the Internet via satellite), MEC, and related technologies attempt to provide reduced latency, increased responsiveness, and greater computing power compared to traditional cloud network services and WAN connections. However, the integration of mobility and dynamically launched services introduces constraints and concerns regarding orchestration, function coordination, and resource management for certain mobile use and device handling use cases, especially in complex mobility settings involving many stakeholders (e.g., devices, hosts, tenants, service providers, operators, etc.).

[0027] Similarly, Internet of Things (IoT) networks and devices are designed to provide distributed computing deployments from a variety of endpoints. IoT devices can be physical or virtual objects communicating on a network and can include sensors, actuators, and other input / output components. IoT devices can be used to collect data or perform actions in a real-world environment. For example, IoT devices can include low-power endpoint devices embedded in or attached to everyday objects such as buildings, vehicles, and packages to provide an additional level of artificial perception of those objects. IoT devices have become increasingly popular, and consequently, applications using these devices have proliferated.

[0028] In some examples, the edge environment may include the enterprise edge, where communication with and / or within the enterprise edge can be facilitated via wireless and / or wired connectivity. The deployment of various edge, fog, MEC, and IoT networks, devices, and services has introduced numerous advanced use cases and scenarios that occur at and towards the network edge. However, these advanced use cases also introduce corresponding technical challenges related to many other issues such as security, processing and network resources, service availability and efficiency, and more. One such challenge relates to edge, fog, MEC, and IoT networks, devices, and services performing workloads on behalf of endpoint devices, including establishing probabilities to determine data integrity and / or data constraints.

[0029] Existing technologies and configurations can be used in conjunction with many aspects of current networked systems, but are provided with reference to edge cloud, IoT, MEC, and other distributed computing deployments. The following systems and technologies can be implemented in or enhance various distributed, virtualized, or managed edge computing systems. These include environments in which network services are implemented or managed using MEC, fourth-generation (4G), or fifth-generation (5G) wireless network configurations; or environments employing wired network configurations involving fiber optic, copper, and / or other connections. Furthermore, aspects of processing performed by the respective computing components can involve computing elements geographically close to user equipment or other endpoint locations such as smartphones, vehicle communication components, IoT devices, etc. Furthermore, the currently disclosed technologies can relate to other edge / MEC / IoT network communication standards and configurations, as well as other intermediate processing entities and architectures.

[0030] Edge computing is a development paradigm in which computation is performed at or near the “edge” of a network, typically through computing platforms implemented at base stations, gateways, network routers, or other devices closer to the endpoints that generate and consume data. For example, an edge gateway server may be equipped with memory pools and storage resources to perform computation in real time for low-latency use cases (e.g., autonomous driving or video surveillance) of connected client devices. Alternatively, as an example, a base station may be augmented with computing and acceleration resources to directly handle service workloads for connected user equipment without further data transmission via a backhaul network. Or, as yet another example, central office network management hardware can be replaced by computing hardware that performs virtualized network functions, provides computing resources for service execution, and provides consumer-side functionality for connected devices.

[0031] An edge environment includes a network and / or a portion of a network located between a cloud environment and an endpoint environment. The edge environment enables workload computation at the edge of the network. For example, an endpoint device (e.g., a user equipment) may request a nearby base station to compute the workload instead of a central server in the cloud environment. The edge environment includes edge services (e.g., an edge leasing platform (EPH)), which include storage pools, storage resources, and processing resources. In some examples, the edge environment may include Edge as a Service (EaaS), which may include one or more edge services. Edge services perform computations, such as workload execution, on behalf of other edge services, edge nodes (e.g., EPH nodes), endpoint devices, etc. The edge environment facilitates connectivity between producers (e.g., workload executors, edge services) and consumers (e.g., other edge services, endpoint devices).

[0032] Because edge services are closer to endpoint devices than centralized servers in a cloud environment, they enable workload computation with lower latency (e.g., response time) compared to cloud environments. Edge services can also localize workload execution based on geographic location or network layout. For example, an endpoint device might request that a workload be executed in a first geographic region, while the centralized server is located in a second geographic region. The endpoint device could then request that the workload be executed by an edge service located in the first geographic region to comply with enterprise or regulatory restrictions.

[0033] Examples of workloads to be performed in edge environments (e.g., via EaaS, via edge services, on EPH nodes, etc.) include autonomous driving computing, video surveillance, machine learning model execution, and real-time data analytics. Additional examples of workloads include delivering and / or encoding media streams, measuring ad impression rates, object detection in media streams, voice analytics, asset and / or inventory management, and augmented reality processing.

[0034] In some examples, edge services enable workloads to be executed with lower response times compared to servers in a cloud environment and return the results of the executed workloads to endpoint devices. For instance, if the edge service is located closer to the endpoint device on the network than a cloud server, the edge service can respond to workload execution requests from the endpoint device much faster than a cloud server. The endpoint device can request the execution of time-constrained workloads from the edge service instead of the cloud server.

[0035] Furthermore, edge services enable the distribution and decentralization of workload execution. For example, an endpoint device may request a first workload execution and a second workload execution. In some examples, a cloud server may respond to both workload execution requests. However, in an edge environment, a first edge service may execute the first workload execution request, and a second edge service may execute the second workload execution request.

[0036] Additional infrastructure may be included in the edge environment to facilitate the execution of workloads on behalf of endpoint devices. For example, an orchestrator may access requests from endpoint devices to execute workloads and provide offers to multiple edge nodes. These offers may include a description of the workload to be executed and terms regarding energy and resource constraints. Edge nodes (e.g., EPH nodes) may accept offers, execute workloads, and provide the results of the execution to the infrastructure in the edge environment and / or to the endpoint devices.

[0037] In the Edge-as-a-Service (EaaS) ecosystem, service delivery (e.g., in an edge environment, via EPH, via edge infrastructure components, etc.) may include a business model where subscribers of the EaaS service (e.g., endpoint devices, user devices, etc.) pay for access to the edge service. In some examples, endpoint devices may pay for edge services (such as workload execution) via micropayments, credit, tokens, electronic money, etc. In some examples, revenue models may include mobile network operators maintaining subscriptions from a subscriber base (e.g., one or more networks) through Service Level Agreement (SLA) contracts as a means of paying for edge services. For example, an SLA may include one or more Service Level Objectives (SLOs). SLOs may include metrics such as uptime, response time, etc. In some examples, an SLA corresponds to the resources used to achieve a certain type of SLO. For example, an SLA specifies the number of cores, memory bandwidth, etc., to achieve an SLO of 30 frames per second for an artificial intelligence model, etc. Accounting performed and / or managed by the MNO determines the billable services subsequently applied to subscriber accounts.

[0038] In some examples, a single resource or entity can negotiate and manage multiple SLAs in an edge computing environment, while using one or more SLOs to manage its resources. For example, SLOs may be based on an edge cloud environment. SLOs can be instantiated within a service. SLAs are instantiated within an associated resource. Multiple SLAs may compete for a specific resource. In some examples, SLAs can be grouped using associated trust levels. Grouping of SLAs can be dynamic based on requirements, trust, user / device type, etc. For example, a group key can be associated with an SLA.

[0039] The rapid growth of edge computing presents both challenges and opportunities for large-scale software engineering practices. For example, debugging, A / B testing, correlating anomalies, eliminating factors as causes of vulnerabilities and / or failures, and establishing patterns between errors and failures are complex operations across a large number of independently developed microservices that work together to deliver end-user value. Furthermore, these problems are compounded at the network edge, where computing infrastructure is heterogeneous, different systems are maintained in different locations and subject to different fault profiles, timing anomalies are more likely to occur due to non-uniform communication and non-localized placement, and other ambiguities exist (such as power-constrained and bandwidth-constrained task distribution). The fact that different edge locations may belong to different trust boundaries adds to these complexities. Therefore, a series of actions across different microservices in loosely coupled interactions can be transparent, semi-transparent, or opaque when attempting to correlate data and statistics to track software / program code execution, collecting debugging information for software code execution, etc. As used in this document, the terms “microservice,” “service,” “task,” “operation,” and “function” are used interchangeably to refer to applications, processes, and / or other software code (also known as program code) that are executed using computing infrastructure such as edge computing environments.

[0040] The examples disclosed herein provide delegated orchestration for managing and orchestrating services across entities in cloud or edge environments. An orchestrator in a cloud and / or edge environment is a resource manager for an application or other executable service. The orchestrator manages the interconnection and interaction between services and / or resources in the cloud / edge environment. The orchestrator may manage resources, distribute workloads, and / or otherwise help ensure the proper execution of services (e.g., according to SLAs, etc.). Some examples provide delegated orchestration across multi-edge architectures, including a set of distinct end-to-end edge deployments belonging to different entities and orchestrated separately. Some examples enable edge owners to maintain control over the scheduling and management of a service throughout its lifecycle, even after the service has been committed to another edge entity. In some examples, multiple orchestrators form a mesh for managing resources used to execute services across a multi-edge architecture. For example, an orchestrator may further control an orchestrator mesh.

[0041] Edge cloud systems can be deployed within an orchestration hierarchy (e.g., within an orchestration domain). At large scale, the orchestration hierarchy of domains can lead to bottlenecks in the orchestration functions performed by the system. For example, complex orchestration functions may involve machine learning, which, when deployed in an edge computing environment, can reduce latency, save bandwidth, improve privacy, and enable smarter applications. However, the orchestration domain hierarchy may diminish or eliminate these benefits and make it difficult for machine learning models to operate across orchestration domains. Some examples allow orchestrators to become part of an “orchestration mesh” rather than requiring each orchestration domain to push down control based on a fixed hierarchy. In these examples, orchestration functions are dynamically organized and can be passed, propagated, or transferred from one orchestrator to another.

[0042] Therefore, orchestrators within multiple orchestrators in an edge cloud computing environment can be used to coordinate services and / or control containers of such services. In some examples, services can be dynamically moved to different edge locations within the edge cloud environment, and associated orchestrators(s) can move with the services between edge compute nodes and / or other locations to facilitate trusted, stable execution of the services. Because services (and associated orchestrators(s)) migrate across compute resources within the edge cloud environment, these services can be managed using orchestration application programming interfaces (APIs) and associated open stack clusters. This allows orchestration privileges to be dynamically distributed across available edge cloud regions and security and trust levels to be maintained across orchestrator meshes, clusters, or groups.

[0043] For example, in order to perform a service and meet SLAs and / or associated SLOs, a server can move the service from one edge or cloud computing resource to another to satisfy SLA and / or SLO requirements. For example, an orchestrator mesh or other orchestrator cluster (e.g., coordinated by orchestrators in the orchestrator mesh, etc.) can be used to transfer control of the service from the first orchestrator to the second orchestrator when the service is moved from the first resource to the second resource for execution.

[0044] In some examples, the ontology provides the orchestrator with distributable functionality to ensure cross-platform consistency, enabling resource X at node 1 to replace resource Y at node 2. For example, resource X could be a high-end field-programmable gate array (FPGA) with high cost and high demand, while resource Y could be a graphics processing unit (GPU)-based software library that delivers lower cost, smaller queuing times, but lower overall processing efficiency. The service ontology can help determine whether a service can use resource X to replace resource Y and can help the orchestrator make such decisions quickly in the hardware.

[0045] Therefore, the management domain can be connected to edge devices, and edge devices and / or other resources can be connected to share requirements, workflows, and / or workload information, thereby meeting SLAs. Connected devices and associated orchestrators in the management domain can negotiate to transfer or "hand over tasks" so that one resource completes a subset of tasks and another resource completes another subset of tasks, thereby distributing work (e.g., based on tokens passed between resources), until the SLA is achieved.

[0046] Figure 1 The example usage environment 100 is described, including an example cloud environment 105, an example edge environment 110, and an example endpoint environment 112. Services can be executed and / or otherwise exist in one or more of the example cloud environment 105, example edge environment 110, and example endpoint environment 112. The cloud environment 105 includes a first example server 115, a second example server 120, a third example server 125, and an example data storage 130. Servers 115, 120, and 125 can execute centralized applications (e.g., website hosting, data management, machine learning model applications, responding to requests from client devices, etc.). Data storage 130 can store information such as database records, website requests, machine learning models, results of executing machine learning models, results of other service executions, traces, code, etc.

[0047] although Figure 1 The example shows three servers 115-125 in cloud environment 105, but cloud environment 105 may include any number of servers. Similarly, although cloud environment 105 includes one data store, it may include any number of data stores. Servers 115, 120, and 125 may communicate with devices in edge environment 110 and / or endpoint environment 112 via a network such as the Internet. Database 130 may provide and / or store data records in response to requests from devices in cloud environment 105, edge environment 110, and / or endpoint environment 112.

[0048] Edge environment 110 (e.g., edge infrastructure) includes a first example edge service 135 and a second example edge service 140. Figure 1In the illustrated example, a first edge service 135 corresponds to a first EPH, and a second edge service 140 corresponds to a second EPH. The first edge service 135 includes an example orchestrator 142, an example scheduler 146, and an example edge node 148 corresponding to the first example EPH node. The second edge service 140 includes an example orchestrator 150, an example blockchain node 152, an example scheduler 154, and an example edge node 156 corresponding to the second example EPH node. Edge services 135 and 140 communicate with each other and with services 115, 120, and 125 in the cloud environment 105. As used herein, the phrase "to communicate" (including its various variations) includes direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or continuous communication, but additionally includes selective communication performed at periodic intervals, predetermined intervals, non-periodic intervals, and / or one-off events. Although the example orchestrators 142, 150 and the example schedulers 146, 154 are shown as being implemented in the example edge environment 110, the orchestrators 142, 150 and / or schedulers 146, 154 may exist in the cloud environment 105, the edge environment 110 and / or the endpoint environment 112.

[0049] Edge services 135 and 140 may perform workloads on behalf of devices located in cloud environment 105, edge environment 110, and / or endpoint environment 112. Edge services 135 and 140 may communicate with devices (e.g., first server 115, data storage 130, etc.) in environments 105, 110, and 112 via a network (such as the Internet). Further, edge service 135 may communicate wirelessly (e.g., via cellular communication, satellite communication, etc.) with elements in environments 105, 110, and 112. For example, edge service 135 may connect to (e.g., communicate with) a cellular base station included in cloud environment 105 and connected to first server 115. As used herein, the phrase “connected to” (including variations thereof) encompasses direct and / or indirect communication between connected devices.

[0050] Endpoint environment 112 includes a first example endpoint device 165, a second example endpoint device 170, a third example endpoint device 175, a fourth example endpoint device 180, a fifth example endpoint device 185, and a sixth example endpoint device 190. Figure 1In the illustrated example, first endpoint device 165, second endpoint device 170, and third endpoint device 175 are connected to first edge service 135. Similarly, in the illustrated example, fourth endpoint device 180, fifth endpoint device 185, and sixth endpoint device 190 are connected to second edge service 140. However, endpoint devices 165, 170, 175, 180, 185, and 190 can be connected to any number of edge services, servers (e.g., servers 115, 120, 125), and / or any other suitable devices included in environments 105, 110, and 112. For example, first endpoint device 165 can be connected to edge services 135 and 140 and to second server 120. Any of endpoint devices 165, 170, 175, 180, 185, and 190 can be connected to devices in environments 105, 110, and 112 via a network such as the Internet. In some examples, endpoint devices 165, 170, 175, 180, 185, and 190 may connect to one or more cellular base stations included in one of environments 105, 110, and 112. For example, the first endpoint device 165 may connect to a cellular base station included in edge environment 110, and that cellular base station may connect to a first edge service 135. Further, any number of endpoint devices may be included in endpoint environment 112. In some examples, one or more of endpoint devices 165, 170, 175, 180, 185, and 190 may be user devices such as smartphones, personal computers, tablets, etc.

[0051] In response to a request to execute a workload from an endpoint device (e.g., endpoint device 165), an orchestrator (e.g., orchestrator 142) communicates with at least one edge node (e.g., edge node 148) and the endpoint device (e.g., endpoint device 165) to create a contract (e.g., SLA) associated with a description of the workload to be executed. For example, orchestrator 142 may communicate with one or more other orchestrators and / or with a master orchestrator to execute all or part of the workload. Edge node 148 provides orchestrator 142 with an execution environment for the task associated with the contract and the description of the workload to be executed. A scheduler (e.g., scheduler 146) provides resources for a service (e.g., edge service 135) running relative to edge node 148 to execute the task while satisfying the requirements of edge device 165 (e.g., service level agreement requirements, etc.). The task may include a contract and a description of the workload to be executed. In some examples, the task may include tokens (such as micropayments, electronic money, etc.) for acquiring the resources used to execute the workload. In some examples, a task may include a count of tokens to be provided in response to the execution of a workload.

[0052] In some examples, transactions, traces, error detection, and repairs can be stored and tracked in a distributed ledger such as a blockchain. Blockchain nodes (e.g., blockchain node 152) maintain records and / or logs of actions occurring in environments 105, 110, and 112. In some examples, blockchain 152 may include a Merkle tree, and blockchain node 152 can provide proof of delivery of contracts, bids, offers, results, etc., to parties involved in a transaction (such as endpoint device 165, orchestrator 142, edge node 148, and / or any device included in environments 105, 110, and 112). For example, edge node 148 can notify blockchain node 152 of the receipt of a workload description. Orchestrators 142 and 150, workload schedulers 146 and 154, and / or edge nodes 148 and 156 can provide records of actions and / or tokens to blockchains 144 and 152. For example, orchestrator 142 can provide records to blockchain nodes(s)152 for receiving requests to execute workloads (e.g., contract requests provided by endpoint device 165). For example, blockchain 152 can be implemented as a centralized server in cloud environment 105. Further, any device in environment 100 (e.g., endpoint device 165) can provide records and / or tokens to blockchain node 152. For example, endpoint device 165 can accept contracts (such as SLAs, electronic contracts, etc.) provided by orchestrator 142 and provide notification of contract acceptance to blockchain node 152. In some examples, blockchain node 152 can be implemented as a service controlled by a centralized server (e.g., first server 115).

[0053] Scheduler 146 accesses a task from orchestrator 142 and provides the task to edge node 148. Edge node 148 executes the workload based on a description of the workload included in the task. Scheduler 146 accesses the results and / or tokens of the workload execution from edge node 148. Scheduler 146 provides the results to endpoint device 165. In some examples, scheduler 146 retains a portion of the tokens accessed from edge node 148 and / or provides these tokens to orchestrator 142, endpoint device 165, and / or resource provider.

[0054] Edge node 148 accesses tasks from scheduler 146 and executes the workload based on the description of the work conforming to the task. Edge node 148 provides at least one execution result of the workload to scheduler 146 and distributes tokens to resource providers (such as energy providers and / or EPH providers). In some examples, the EPH provider provides infrastructure services (e.g., maintenance, equipment support, data center services, space allocation, etc.) to the EPH node. In some examples, edge nodes 148 and 156 may provide tokens to schedulers 146 and 154 in response to determining that contract terms associated with the execution of the workload have not been satisfied. In some examples, in response to determining that contract terms have not been satisfied, edge nodes 148 and 156 may request additional tokens from schedulers 146 and 154, orchestrators 142 and 150, and / or endpoint device 165.

[0055] exist Figure 1 In the example illustrated, orchestrator 142, scheduler 146, and edge node 148 are included in edge service 135. However, in some examples, orchestrator 142, scheduler 146, and / or edge node 148 are included in edge environment 110 instead of edge service 135. For example, orchestrator 142 may be connected to cloud environment 105 and / or endpoint environment 112 located outside edge service 135. In another example, orchestrator 142, scheduler 146, and / or edge node 148 are separate devices included in edge environment 110. Further, any one of orchestrator 142, scheduler 146, or edge node 148 may be included in cloud environment 105 or endpoint environment 112. For example, orchestrator 142 may be included in endpoint environment 112, in first service 115 in cloud environment 105, and so on. In some examples, scheduler 146 may be included in orchestrator 142 instead of edge service 135. In some examples, edge services 135, 140 may be distributed and executed among two or more peer devices (such as two or more edge services) included in edge environment 110.

[0056] Executing a workload in an edge environment 110 can reduce the computational cost and / or processing time required to execute the workload compared to executing it in the cloud environment 105. For example, endpoint devices 165-190 can request edge services 135, 140 to execute the workload at a lower cost than it would require to execute the workload in the cloud environment 105. In some examples, multiple edge services 135, 140 can compete to accept tasks to execute the workload, and each edge service 135, 140 can submit a bid to orchestrator 142 and / or endpoint device 165, including the workload execution cost. Each bid can include a different workload execution cost, and edge services 135, 140 can reduce their respective workload execution costs relative to other bids to be selected for workload execution. Thus, the workload execution cost in the edge environment 110 may be lower than the workload execution cost provided by a centralized server (e.g., the first server 115) in the cloud environment 105.

[0057] In another example, endpoint devices 165-190 can be located closer to edge services 135 and 140 than to the centralized server in cloud environment 105. For example, edge service 135 is closer to endpoint device 165 than to the first server 115. As a result, endpoint device 165 can request edge service 135 to perform a workload, and the response time of edge service 135 in delivering the results of the performed workload is lower than the response time of the first server 115 in cloud environment 105.

[0058] Consistent with the examples provided herein, endpoint devices (e.g., one of endpoint devices 165, 170, 175, 180, 185, 190) can be implemented as any type of endpoint component, device, apparatus, or other thing capable of communicating as a producer or consumer of data. For example, endpoint devices may include mobile phones, laptop computers, desktop computers, processor platforms in autonomous vehicles, etc. In additional or alternative examples, endpoint devices may include cameras, sensors, etc. Furthermore, the labels “platform,” “node,” and / or “device” as used in environment 100 do not necessarily refer to such platforms, nodes, and / or devices operating in a client or subordinate role; rather, any of the platforms, nodes, and / or devices in environment 100 refers to various entities, platforms, nodes, devices, and / or subsystems including discrete and / or connected hardware and / or software configurations to facilitate and / or use of edge environment 110.

[0059] In some examples, edge environment 110 is formed from network components and functional features (e.g., orchestrator 142, edge node 148, etc.) operated by and residing within edge service 135. Edge environment 110 can be implemented as any type of network providing edge computing and / or storage resources located close to endpoint devices (e.g., mobile computing devices, IoT devices, smart devices, etc.) supporting a radio access network (RAN). Figure 1 The endpoint devices 165, 170, 175, 180, 185, and 190 are shown in the diagram. In other words, the edge environment 110 can be envisioned as the “edge” connecting the endpoint devices 165, 170, 175, 180, 185, and 190 and traditional network access points. This “edge” acts as an entry point into the core network of a service provider, including mobile operator networks (e.g., Global System for Mobile Communications (GSM) networks, Long Term Evolution (LTE) networks, 5G and / or 6G networks, etc.), while also providing storage and / or computing capabilities. Other types and forms of network access (e.g., Wi-Fi, long-range wireless, wired networks including optical networks) may also be utilized in lieu of or in combination with such 3GPP operator networks.

[0060] Figure 2 Block diagram 200 illustrates an overview of another configuration for edge computing, which includes a processing layer referred to as an "edge cloud" in many of the following examples. As shown, edge cloud 210 is co-located at an edge location (such as an access point or base station 240, a local processing hub 250, or a central office 220) and can therefore include multiple instances of entities, devices, and equipment. Compared to cloud data center 230, edge cloud 210 is located closer to endpoint (consumer and producer) data sources 260 (e.g., autonomous vehicles 261, user equipment 262, commercial and industrial equipment 263, video capture equipment 264, drones 265, smart city and building equipment 266, sensors and IoT devices 267, etc.). The computing, memory, and storage resources provided at the edge in edge cloud 210 are crucial for providing ultra-low latency response times for services and functions used by endpoint data sources 260 and for reducing network backhaul traffic from edge cloud 210 to cloud data center 230, thus improving benefits such as energy consumption and overall network usage.

[0061] Computation, memory, and storage are scarce resources and typically decrease with edge location (e.g., fewer processing resources are available on a consumer endpoint device than on a base station or at a central office). However, the closer an edge location is to the endpoint (e.g., the UE), the more limited its space and power tend to be. Therefore, edge computing attempts to reduce the amount of resources required for network services by allocating more resources that are located both geographically and temporally closer to the network. In this way, edge computing attempts to bring computing resources to workload data, or vice versa, where appropriate.

[0062] The following describes various aspects of the edge cloud architecture, which encompasses multiple potential deployments and addresses limitations that network operators or service providers may have within their own infrastructure. These include configuration variations based on edge location (e.g., because the edge at the base station level may have more limited performance and capabilities in multi-tenant scenarios); configurations based on the types of compute, memory, storage, architecture, acceleration, and other resources available at the edge location, the layer of location, or the group of locations; service, security, and management and orchestration capabilities; and related objectives for achieving availability and performance of end services. Depending on latency, distance, and timing characteristics, these deployments can be handled at network layers, which can be viewed as “near-edge,” “close-to-edge,” “local-edge,” “intermediate-edge,” or “far-edge” layers.

[0063] Edge computing is a development paradigm in which computation is performed at or near the “edge” of a network, typically using computing platforms (e.g., x86 or ARM computing hardware architectures) implemented at base stations, gateways, network routers, or other devices closer to the endpoints that generate and consume data. For example, an edge gateway server may be equipped with memory pools and storage resources to perform computation in real time for low-latency use cases (e.g., autonomous driving or video surveillance) of connected client devices. Alternatively, as an example, a base station may be augmented with computing and acceleration resources to directly handle service workloads for connected user equipment without further data transmission via a backhaul network. Or, as another example, central office network management hardware can be replaced by standardized computing hardware that performs virtualized network functions, provides computing resources for service execution, and provides consumer functions for connected devices. Within an edge computing network, there may be scenarios where computing resources are “moved” to services containing data, and scenarios where data is “moved” to computing resources. Alternatively, as an example, base station computing, acceleration, and network resources can be provided to scale workload demands as needed by activating dormant capacity (subscription, on-demand capacity) to manage extreme situations, emergencies, or to provide long lifespans for deployed resources over a significantly longer implementation lifecycle.

[0064] Figure 3 The diagram illustrates the operational layer between endpoints, edge cloud, and cloud computing environments. Specifically, Figure 3 An example of computing use case 305 utilizing edge cloud 210 across multiple illustrative layers of network computing is depicted. These layers begin with endpoint (device and thing) layer 300, which accesses edge cloud 210 for data creation, analysis, and data consumption activities. Edge cloud 210 may span multiple network layers (such as edge device layer 310 with gateways, on-premise servers, or network devices (nodes 315) located in physically adjacent edge systems); network access layer 320, which encompasses base stations, radio processing units, network hubs, regional data centers, or local network equipment (equipment 325); and any equipment, devices, or nodes located in between (not shown in detail in layer 312). Network communication within edge cloud 210 and between layers can be achieved via any number of wired or wireless media, including via connectivity architectures and technologies not depicted.

[0065] Examples of latency due to network communication distance and processing time constraints range from less than one millisecond (ms) at endpoint layers 300, less than 5 ms at edge device layers 310, to even 10 to 40 ms when communicating with nodes at network access layers 320. Beyond the edge cloud 210 are the core network layer 330 and the cloud data center layer 340, each with increased latency (e.g., 50-60 ms at the core network layer 330, to 100 ms or more at the cloud data center layer). Therefore, operations with latency of at least 50 to 100 ms or more at the core network data center 335 or cloud data center 345 will be unable to perform many time-critical functions of use case 305. Each of these latency values ​​is provided for illustrative and comparative purposes; it should be understood that latency can be further reduced using other access network media and technologies. In some examples, different parts of a network can be categorized into “near-edge”, “local”, “near-edge”, “intermediate”, or “far-edge” layers relative to the network source and destination. For example, from the perspective of core network data center 335 or cloud data center 345, the central office or content data network can be considered to be located within the “near-edge” layer (“close” to the cloud, with high latency values ​​when communicating with devices and endpoints in use case 305), while access points, base stations, internal servers, or network gateways can be considered to be located within the “far-edge” layer (“far” from the cloud, with low latency values ​​when communicating with devices and endpoints in use case 305). It should be understood that other classifications of specific network layers constituting “near,” “local,” “near,” “intermediate,” or “far” edges can be based on latency, distance, network hop count, or other measurable characteristics, such as those measured from sources in any of network layers 300-340.

[0066] Because multiple services utilize the edge cloud, various use cases 305 can access resources under the pressure of usage from incoming flows. To achieve low latency results, the services executing within the edge cloud 210 balance different requirements in the following ways: (a) priority (throughput or latency) and quality of service (QoS) (e.g., in terms of response time requirements, the traffic of an autonomous vehicle may have a higher priority than that of a temperature sensor; or, depending on the application, performance sensitivity / bottleneck may exist on compute / accelerator, memory, storage, or network resources); (b) reliability and resilience (e.g., depending on the application, some input flows need to be acted upon and traffic routed with mission-critical reliability, while some other input flows can tolerate occasional failures); and (c) physical constraints (e.g., power, cooling, and form factor).

[0067] These use cases’ end-to-end service views involve the concept of service flow and are associated with transactions. A transaction details the overall service requirements of the entity consuming the service, as well as the associated services in terms of resources, workloads, workflows, and business functions and business-level requirements. Services executed according to the described “Terms” are managed at each layer in a way that ensures the real-time and runtime contract compliance of the transaction throughout the service’s lifecycle. When a component in a transaction misses its agreed SLA, the system as a whole (the components in the transaction) can provide the following capabilities: (1) understanding the impact of an SLA breach and (2) enhancing other components in the system to restore the overall transaction SLA and (3) implementing remedial steps.

[0068] Therefore, considering these changes and service characteristics, edge computing within the edge cloud 210 can provide services and responses to multiple applications in use case 305 (e.g., object tracking, video surveillance, connected cars, etc.) in real-time or near real-time, while meeting the ultra-low latency requirements of these applications. These advantages enable entirely new categories of applications (Virtual Network Functions (VNFs), Function as a Service (FaaS), Edge as a Service (EaaS), standard processes, etc.) that cannot leverage traditional cloud computing due to latency or other limitations.

[0069] However, along with the advantages of edge computing come the following considerations. Devices located at the edge are often resource-constrained, and therefore there is pressure on the use of edge resources. This is typically addressed by pooling memory and storage resources for use by multiple users (tenants) and devices. The edge may be power- and cooling-constrained, and therefore power usage needs to be handled by the applications that consume the most power. There may be inherent power performance trade-offs in these pooled memory resources, as many may use emerging memory technologies in which more power is used to deliver greater memory bandwidth. Similarly, improved hardware security and trusted root functionality are also required, as edge locations can be unmanned (controlled) and may even require authorized access (e.g., when housed in a third-party location). These issues are amplified in multi-tenant, multi-owner, or multi-access setups, where services and applications are requested by many users, especially when network usage fluctuates dynamically and the composition of multiple stakeholders, use cases, and services changes.

[0070] At a more general level, an edge computing system can be described as encompassing any number of deployments at the layers previously discussed that operate within the edge cloud 210 (network layers 300-340), providing coordination from clients and distributed computing devices. One or more edge gateway nodes, one or more edge aggregation nodes, and one or more core data centers can be distributed across the various layers of the network to provide, or represent, the implementation of the edge computing system by a telecommunications service provider (“Telecommunications Company” or “TSP”), an IoT service provider, a cloud service provider (CSP), an enterprise entity, or any other number of entities. Various implementations and configurations of the edge computing system can be dynamically provided, such as when orchestrating to meet service objectives.

[0071] Consistent with the examples provided herein, client computing nodes can be embodied as any type of endpoint component, device, apparatus, or other thing capable of communicating as a producer or consumer of data. Furthermore, the labels “node” or “device” used in edge computing systems do not necessarily refer to such nodes or devices operating in a client or agent / servant / follower role; rather, any of the nodes or devices in an edge computing system refers to various entities, nodes, or subsystems including discrete and / or connected hardware or software configurations to facilitate and / or utilize the edge cloud 210.

[0072] Thus, the edge cloud 210 is formed by network components and functional characteristics operated by edge gateway nodes and edge aggregation nodes or other edge computing nodes in network layers 310-330 and operated within edge gateway nodes and edge aggregation nodes or other edge computing nodes in network layers 210-230. Therefore, the edge cloud 210 can be embodied as any type of network providing edge computing and / or storage resources located close to endpoint devices (e.g., mobile computing devices, IoT devices, smart devices, etc.) supporting the radio access network (RAN), as discussed herein. In other words, the edge cloud 210 can be envisioned as an “edge” connecting endpoint devices and traditional network access points while also providing storage and / or computing capabilities, acting as an entry point into service provider core networks, including mobile operator networks (e.g., GSM networks, LTE networks, 5G / 6G networks, etc.). Other types and forms of network access (e.g., Wi-Fi, long-range wireless, wired networks including optical networks) may also be used in place of or in combination with such 3GPP operator networks.

[0073] The network components of the edge cloud 210 may include servers, multi-tenant servers, device computing devices, and / or any other type of computing device. For example, the edge cloud 210 may be a device computing device as a self-contained electronic device including a housing, chassis, enclosure, or shell. In some cases, the housing size may be determined for portability so that it can be carried and / or transported by humans. Example housings may include materials forming one or more outer surfaces that partially or completely protect the contents of the device, wherein protection may include weather protection, hazardous environment protection (e.g., EMI, vibration, extreme temperatures), and / or enable immersion in water. Example housings may include power circuitry for providing power for stationary and / or portable implementations, such as AC power input, DC power input, (multiple) AC / DC or DC / AC converters, power conditioners, transformers, charging circuitry, batteries, wired inputs, and / or wireless power inputs. Example housings and / or their surfaces may include or be connected to mounting hardware to enable attachment to structures such as buildings, telecommunications structures (e.g., poles, antenna structures, etc.) and / or racks (e.g., server racks, blade racks, etc.). Example housings and / or their surfaces may support one or more sensors (e.g., temperature sensors, vibration sensors, light sensors, acoustic sensors, capacitive sensors, proximity sensors, etc.). One or more such sensors may be contained in, carried by, or otherwise embedded in and / or mounted to the surface of the device. Example housings and / or their surfaces may support mechanical connections, such as propulsion hardware (e.g., wheels, propellers, etc.) and / or articulated hardware (e.g., robotic arms, pivotable attachments, etc.). In some cases, sensors may include any type of input device, such as user interface hardware (e.g., buttons, switches, dial pads, sliders, etc.). In some cases, example housings may include output devices contained therein, carried therein, embedded therein, and / or attached thereto. Output devices may include displays, touchscreens, lights, LEDs, speakers, I / O ports (e.g., USB), etc. In some cases, edge devices are devices that exist in a network for a specific purpose but may have processing or other capabilities that can be used for other purposes (e.g., traffic lights). Such edge devices can exist independently of other networked devices and are housed in a form factor suited to their primary purpose; however, they remain usable for other computing tasks that do not interfere with their primary task. Edge devices include Internet of Things (IoT) devices. Device computing devices may include hardware and software components for managing local issues such as device temperature, vibration, resource utilization, updates, power issues, physical and network security. Figure 8BExample hardware for implementing device computing devices is described. Edge cloud 210 may also include one or more servers and / or one or more multi-tenant servers. Such servers may include operating systems and virtual computing environments. Virtual computing environments may include hypervisors that manage (build, deploy, destroy, etc.) one or more virtual machines, one or more containers, etc. Such virtual computing environments provide an execution environment in which one or more applications and / or other software, code, or scripts can be executed in isolation from one or more other applications, software, code, or scripts.

[0074] Figure 4 The diagram illustrates a block diagram of an example environment 400 in which various client endpoints 410 (in the form of mobile devices, computers, autonomous vehicles, business computing equipment, and industrial processing equipment) exchange requests and responses with an example edge cloud 210. For example, computers, business computing equipment, and industrial processing equipment can obtain network access via an internal network system 432 to exchange requests and responses 422, or via a wired broadband network. Mobile computing devices can obtain network access via a cellular network tower 434 to exchange requests and responses 424, or via a wireless broadband network. Autonomous vehicles can obtain network access via a street positioning network system 436 to exchange requests and responses 426 via a wireless vehicular network. However, regardless of the type of network access, the TSP can deploy aggregation points 442, 444 within the edge cloud 210 to aggregate traffic and requests. Therefore, within the edge cloud 210, the TSP can deploy various computing and storage resources (such as at edge aggregation node 440) to provide the requested content. Edge aggregation node 440 and other systems in edge cloud 210 are connected to cloud or data center 460, which uses backhaul network 450 to fulfill complex but more latency-tolerant requests from the cloud / data center to websites, applications, database servers, etc. Additional or merged instances of edge aggregation node 440 and aggregation points 442, 444 (including those deployed on a single server frame) may also exist in other areas of edge cloud 210 or the TSP infrastructure.

[0075] Figure 5 The diagram illustrates the deployment and orchestration of a virtual edge configuration for an edge computing system operating across multiple edge nodes and multiple tenants. Specifically, Figure 5The coordination of a first edge node 522 and a second edge node 524 in an edge computing system 500 is depicted to meet and respond to requests from various client endpoints 510 (e.g., smart city / building systems, mobile devices, computing devices, commercial / logistics systems, industrial systems, etc.) accessing various virtual edge instances. Here, virtual edge instances provide edge computing capabilities and processing in an edge cloud that accesses a cloud / data center 540 to obtain higher latency requests for websites, applications, database servers, etc. However, the edge cloud is capable of coordinating processing between multiple edge nodes of multiple tenants or entities.

[0076] exist Figure 5 In the example, these virtual edge instances include: a first virtual edge 532 provided to a first tenant (tenant 1), which provides a first combination of edge storage, computing, and services; and a second virtual edge 534 providing a second combination of edge storage, computing, and services. Virtual edge instances 532 and 534 are distributed among edge nodes 522 and 524, and may include scenarios where requests and responses are fulfilled from the same or different edge nodes. The configuration of edge nodes 522 and 524 for operating in a distributed but coordinated manner occurs based on edge provisioning functionality 550. The functionality of edge nodes 522 and 524 for providing coordinated operations for applications and services across multiple tenants occurs based on orchestration functionality 560.

[0077] It should be understood that some of the devices in device 510 that interact with nodes 522 and 524 in edge cloud 500 are multi-tenant devices, where tenant 1 can run within a tenant 1 “slice”, while tenant 2 can run within a tenant 2 “slice” (and, in further examples, there may be additional tenants or subtenants; and each tenant may even have specific rights and be transactionally bound to a particular group of features, down to specific hardware features). Trusted multi-tenant devices may further include tenant-specific cryptographic keys, such that the combination of keys and slices can be considered a “root of trust” (RoT) or a tenant-specific RoT. The RoT for dynamic computation can be further composed using a DICE (Device Identity Composition Engine) architecture, allowing a single DICE hardware building block to be used to construct a hierarchical trusted computational foundation context for layering device capabilities, such as field-programmable gate arrays (FPGAs). The RoT can further be used in trusted computational contexts to enable “fan-out”, useful for supporting multi-tenancy. Within a multi-tenant environment, the corresponding edge nodes 522 and 524 can serve as points of implementation for security features of local resources allocated to multiple tenants per node. Additionally, tenant runtime and application execution (e.g., in instances 532, 534) can be used as implementation points for security features that create virtual edge abstractions of resources spanning potentially multiple physical supervisor platforms. Finally, orchestration functionality 560 at the orchestration entity can operate as an implementation point for security features used to queue resources along tenant boundaries.

[0078] Edge computing nodes can be partitioned with resources (memory, CPU, GPU, interrupt controller, I / O controller, memory controller, bus controller, etc.), and the corresponding partitions may include RoT capabilities. Furthermore, based on the fan-out and layering of the DICE model, these partitions can be applied to the edge nodes. Cloud computing nodes composed of containers, FaaS engines, small service applications, servers, or other computing abstractions can be partitioned according to the DICE layering and fan-out structure to support the RoT context for each node. Therefore, the corresponding RoT across devices 510, 522, and 540 can coordinate the establishment of a Distributed Trusted Computing Foundation (DTCB), enabling the creation of tenant-specific virtual trusted secure channels that link all elements end-to-end.

[0079] Furthermore, it will be understood that containers may have data- or workload-specific keys that protect their contents from previous edge nodes. As part of container migration, the repository controller at the source edge node can obtain a migration key from the repository controller at the target edge node, where the migration key is used to wrap the container-specific key. When the container / repository migrates to the target edge node, the unwrapping key is exposed to the repository controller, which then decrypts the wrapped key. The key can now be used to perform operations on container-specific data. Migration functionality can be gated by appropriately authenticated edge nodes and repository managers (as described above).

[0080] In a further example, edge computing systems are extended to provide orchestration of multiple applications by using containers (enclosed, deployable software units that provide code and required dependencies) in multi-owner, multi-tenant environments. Multi-tenant orchestrators can be used to perform key management, trust anchor management, and other functions. Figure 5 The concept of trusted 'slices' in edge computing includes provisioning and lifecycle-related security features. For example, an edge computing system can be configured to meet requests and responses from various client endpoints across multiple virtual edge instances (and from the cloud or remote data centers). The use of these virtual edge instances can simultaneously support multiple tenants and multiple applications (e.g., augmented reality (AR) / virtual reality (VR), enterprise applications, content delivery, gaming, compute migration). Furthermore, various types of applications may reside within a virtual edge instance (e.g., general applications; latency-sensitive applications; latency-critical applications; user-plane applications; networking applications, etc.). Virtual edge instances can also span systems owned by multiple owners in different geographical locations (or corresponding computing systems and resources jointly owned or managed by multiple owners).

[0081] For example, each of edge nodes 522, 524 can implement the use of containers, such as the use of container "cabins" 526, 528 that provide a set of one or more containers. In a setup using one or more container cabins, a cabin controller or orchestrator is responsible for the local control and orchestration of the containers within the cabin. Various edge node resources (e.g., storage, compute, services depicted in hexagons) provided for the corresponding edge slices 532, 534 are partitioned according to the needs of each container.

[0082] Once container cabins are used, cabin controllers oversee the partitioning and allocation of containers and resources. Cabin controllers receive instructions from orchestrators (e.g., orchestrator 560) on how best to partition physical resources and for what duration, such as receiving key performance indicator (KPI) targets based on SLA contracts. Cabin controllers determine which containers need which resources and how long it will take to complete the workload and meet the SLA. Cabin controllers also manage container lifecycle operations such as: creating containers, providing resources and applications to containers, coordinating intermediate results among multiple containers working together on a distributed application, and tearing down containers when the workload is complete. Additionally, cabin controllers can act as security mechanisms, preventing resource allocation before the correct tenant is authenticated, or preventing the provision of data or workloads to containers until the authentication result is met.

[0083] Furthermore, by using container cabins, tenant boundaries can still exist, but within the context of each cabin within the container. If each tenant-specific cabin has a tenant-specific cabin controller, a shared cabin controller will exist to merge resource allocation requests, avoiding typical resource shortage scenarios. Further controls can be provided to ensure the authentication and trustworthiness of cabins and cabin controllers. For example, orchestrator 560 can provide authentication verification policies to the local cabin controller performing authentication verification. If authentication satisfies the policy of the first tenant cabin controller but not the second tenant cabin controller, the second cabin can migrate to another edge node that does indeed satisfy the policy. Alternatively, the first cabin can be allowed to execute, and a different shared cabin controller can be installed and invoked before the second cabin executes.

[0084] Figure 6 Additional compute arrangements for deploying containers in an edge computing system are illustrated. As a simplified example, system arrangements 610 and 620 describe where a cabin controller (e.g., container managers 611, 621, and container orchestrator 631) is adapted to launch containerized cabins, functions, and function-as-a-service instances via compute node (615 in arrangement 610), or to individually execute containerized virtualized networking functions via compute node (623 in arrangement 620). This arrangement is adapted to use multiple tenants in system arrangement 630 (using compute node 637), where containerized cabins (e.g., cabin 612), functions (e.g., function 613, VNFs 622, 636), and function-as-a-service instances (e.g., FaaS instance 615) are launched (in addition to executing virtualized networking functions) within virtual machines dedicated to the respective tenants (e.g., VMs 634, 635 for tenants 632, 633). This arrangement is further adapted for use in system arrangement 640, which provides containers 643, 644, or performs various functions, applications and features on compute node 641, as coordinated by container-based orchestration system 541.

[0085] Figure 6 The system layout described provides an architecture that treats VMs, containers, and functions equally in terms of application composition (and the resulting application is a combination of these three components). Each component may involve using one or more accelerator (FPGA, ASIC) components as a local backend. In this way, the application can be partitioned across multiple edge owners, such as by an orchestrator.

[0086] exist Figure 6 In this context, container controllers / managers, container orchestrators, and individual nodes can provide points of security enforcement. However, tenant isolation can be orchestrated where resources allocated to one tenant are different from those allocated to a second tenant, but edge owners collaborate to ensure that resource allocations are not shared across tenant boundaries. Alternatively, resource allocations can be isolated across tenant boundaries because tenants can allow “use” via subscription or transaction / contract-based mechanisms. In these contexts, edge owners can enforce leasing using virtualization, containerization, enclaves, and hardware partitioning schemes. Other isolated environments can include: bare metal (dedicated) equipment, virtual machines, containers, virtual machines on containers, or combinations thereof.

[0087] In further examples, software-defined or controlled silicon hardware, along with other configurable hardware aspects, can be integrated with the applications, functions, and services of an edge computing system. Software-defined silicon can be used to ensure that a component fulfills a contractual or service level agreement by leveraging its ability to repair itself or a portion of a workload based on a resource or hardware component (e.g., through upgrades, reconfiguration, or the provision of new features within the hardware configuration itself).

[0088] It should be understood that the edge computing systems and deployments discussed in this article are applicable to a wide range of solutions, services, and / or use cases involving mobility. As an example, Figure 7 This illustrates the implications for implementing edge clouds (such as, Figure 2This example simplifies the use case of mobile access for applications in an edge computing system 700 (edge ​​cloud 210) to facilitate computing and communication in vehicles. In this use case, the corresponding client computing node 710 can be embodied as an in-vehicle computing system (e.g., an in-vehicle navigation and / or infotainment system) located in the corresponding vehicle, communicating with the example edge gateway node 720 while traversing a road. For example, the edge gateway node 720 can be located in a roadside cabinet or in another enclosure built into a structure with other, separate, mechanical utilities, which can be placed along a road, at a road intersection, or elsewhere near the road. As the corresponding vehicle travels along the road, the connection between its client computing node 710 and a specific edge gateway node in the edge gateway node 720 can be propagated to another edge gateway node in the edge gateway node 720 to maintain consistent connectivity and context for the example client computing node 710. Similarly, mobile edge nodes can be aggregated at high-priority services or based on throughput or latency resolution requirements of (multiple) underlying services (e.g., in the case of a drone). The corresponding edge gateway device 720 includes a certain amount of processing and storage capacity, and thus, some data processing and / or data storage of the client computing node 710 can be performed on one or more edge gateway nodes 720.

[0089] Edge gateway node 720 can communicate with one or more edge resource nodes 740, which are illustratively specified as computing servers, devices, or components located at or within a communication base station 742 (e.g., a base station of a cellular network). As discussed above, each edge resource node(s) 740 includes a certain amount of processing and storage capacity, and thus, some data processing and / or data storage of the client computing node 710 can be performed on the edge resource nodes(s) 740. For example, less urgent or less important data processing can be performed by the edge resource nodes(s) 740, while data processing with higher urgency or importance can be performed by the edge gateway device 720 (e.g., depending on the capabilities of each component, or information indicating urgency or importance in the request). Work can continue on the edge resource nodes when processing priorities change during processing activity, based on data access, data location, or latency. Similarly, configurable system or hardware resources themselves can be activated (e.g., via a local orchestrator) to provide additional resources to meet new demands (e.g., to adapt computing resources to workload data).

[0090] Multiple edge resource nodes 740 also communicate with a core data center 750, which may include computing servers, devices, and / or other components located in a central location (e.g., the central office of a cellular communication network). Example core data center 750 may provide a gateway to a global network cloud 760 (e.g., the Internet) for the operation of an edge cloud 210 formed by multiple edge resource nodes 740 and edge gateway devices 720. Additionally, in some examples, core data center 750 may include a certain amount of processing and storage capacity, and therefore, some data processing and / or storage (e.g., low-urgency, low-importance, or high-complexity processing) for client computing devices can be performed on core data center 750.

[0091] Edge gateway node 720 or (multiple) edge resource nodes 740 can provide access to stateful application 732 and geographic distributed database 734. While application 732 and database 734 are illustrated as being distributed horizontally at the edge cloud layer, it will be understood that application resources, services, or other components can be vertically distributed throughout the edge cloud (including portions of the application running at client compute node 710, and others at edge gateway node 720 or (multiple) edge resource nodes 740, etc.). Furthermore, as previously stated, peering relationships at any level can exist to fulfill service objectives and obligations. Further, data for a particular client or application can be moved from the edge to the edge based on changing conditions (e.g., based on accelerating resource availability, following vehicle movement, etc.). For example, predictions can be made based on the “decay rate” of access to identify the next owner to continue, or when data or compute access will no longer be feasible. These and other services can be utilized to accomplish the work required to maintain transaction compliance and non-destructive behavior.

[0092] In a further scenario, container 736 (or a container's pod) can be flexibly migrated from one edge node of edge nodes 720 to other edge nodes (e.g., another edge node of these edge nodes 720, a resource node of (multiple) resource nodes 740, etc.), so that containers with applications and workloads do not need to be reorganized, recompiled, or reinterpreted for migration to work. However, in such a setup, some remedial or "hybrid" transformation operations may be applied. For example, the physical hardware at (multiple) edge resource nodes 740 may differ from the hardware at edge gateway nodes 720, and therefore, the Hardware Abstraction Layer (HAL) that makes up the bottom edge of the container will be remapped to the physical layer of the target edge node. This may involve some form of late-binding technique, such as binary conversion of the HAL from the container's native format to the physical hardware format, or it may involve mapping interfaces and operations. A pod controller can be used to drive interface mapping as part of the container's lifecycle, including migration to / from different hardware environments.

[0093] Figure 7 The scenarios covered can utilize various types of mobile edge nodes (such as edge nodes controlling the system in vehicles (cars / trucks / trams / trains) or other mobile units) because the edge nodes will move to other geographical locations along the platform controlling them. In the case of vehicle-to-vehicle communication, each vehicle can even act as a network edge node for other vehicles (e.g., to perform caching, reporting, data aggregation, etc.). Therefore, it will be understood that the application components provided in various edge nodes can be distributed in static or mobile settings, including some functions or operations at individual endpoint devices or edge gateway nodes 720, some other functions or operations at edge resource nodes (multiple) 740, and coordination between other functions or operations in the core data center 750 or global network cloud 760.

[0094] In further configurations, edge computing systems can implement FaaS computing capabilities by using appropriate executable applications and functions. In the example, developers write function code (e.g., "computer code" in this document) representing one or more computer functions, and this function code is uploaded to a FaaS platform provided by, for example, edge nodes or data centers. Triggers (such as, for example, service use cases or edge processing events) initiate the execution of the function code using the FaaS platform.

[0095] In the FaaS example, containers are used to provide an environment in which functional code (e.g., an application that may be provided by a third party) is executed. Containers can be any isolated execution entity, such as processes, Docker or Kubernetes containers, virtual machines, etc. Within an edge computing system, various data center, edge, and endpoint (including mobile) devices are used for on-demand scaling of “spin-accelerate” functionality (e.g., activating and / or assigning functional actions). The functional code is executed on physical infrastructure devices (e.g., edge computing nodes) and the underlying virtualized containers. Finally, the container is “spin-decelerated” on the infrastructure in response to the completion of execution (e.g., deactivated and / or deallocated).

[0096] Other aspects of FaaS enable the deployment of edge functions as services, including support for corresponding functions that support edge computing as a service (Edge as a Service or "EaaS"). Additional features of FaaS may include: fine-grained billing components that allow customers (e.g., computer code developers) to pay only when their code is executed; a general data store for storing data to be reused by one or more functions; orchestration and management between functions; function execution management, parallelism, and merging; management of container and function memory space; coordination of acceleration resources available for functions; and the distribution of functions among containers (including "warm" containers that are already deployed or operational relative to "cold" containers that require initialization, deployment, or configuration).

[0097] The edge computing system 700 may include or communicate with an edge provisioning node 744. The edge provisioning node 744 can provide services such as... Figure 8B The example computer-readable instruction 882, or similar software, is distributed to various recipients for use in implementing any of the methods described herein. The example edge provisioning node 744 can be implemented by any computer server, home server, content delivery network, virtual server, software distribution system, central facility, storage device, storage node, data facility, cloud service, etc., capable of storing software instructions and / or transmitting software instructions (e.g., code, scripts, executable binaries, containers, packages, compressed files, and / or derivatives thereof) to other computing devices. The components(s) of the example edge provisioning node 744 can be located in the cloud, in a local area network, in an edge network, in a wide area network, on the Internet, and / or in any other location communicatively coupled to the recipients(s). Recipients can be customers, clients, partners, users, etc., of entities that own and / or operate the edge provisioning node 744. For example, the entity owning and / or operating the edge provisioning node 744 can be software instructions (such as...) Figure 8BThe example computer-readable instruction (882) is intended for the developer, seller, and / or licensor (or their customers and / or consumers). Recipients may be consumers, service providers, users, retailers, OEMs, etc., who purchase and / or license the software instructions for use and / or resell and / or sublicense.

[0098] In the example, edge provisioning node 744 includes one or more servers and one or more storage devices. As described below, the storage device master computer-readable instructions, such as... Figure 8B Example computer-readable instruction 882. Similar to the edge gateway device 720 described above, one or more servers of the edge provisioning node 744 communicate with the base station 742 or other network communication entities. In some examples, as part of a business transaction, one or more servers respond to a request to transmit software instructions to a requesting party. Payment for the delivery, sale, and / or licensing of the software instructions may be handled by one or more servers of a software distribution platform and / or via a third-party payment entity. The server enables the purchaser and / or licensor to download computer-readable instructions 882 from the edge provisioning node 744. For example, software instructions (which can be used with...) Figure 8B The example computer-readable instructions 882 (corresponding to which) can be downloaded to one or more example processor platforms for executing the computer-readable instructions 882 to implement the methods described herein.

[0099] In some examples, the processor platforms(s) executing the computer-readable instructions 882 may be physically located in different geographical locations, legal jurisdictions, etc. In some examples, one or more servers of the edge provisioning node 744 periodically provide, transmit, and / or force update software instructions (e.g., Figure 8B Example computer-readable instructions 882 (e.g., ensuring that improvements, patches, updates, etc., are distributed and applied to software instructions implemented at end-user devices). In some examples, different components of the computer-readable instructions 882 may be distributed from different sources and / or to different processor platforms; for example, different libraries, plugins, components, and other types of computing modules, whether compiled or interpreted, may be distributed from different sources and / or to different processor platforms. For example, a portion of the software instructions (e.g., a script that is not executable on its own) may be distributed from a first source, while an interpreter (capable of executing the script) may be distributed from a second source.

[0100] In a further example, any of the computing nodes or devices discussed with reference to current edge computing systems and environments can be based on Figure 8A and Figure 8BThe components described herein are used to implement edge computing nodes. A corresponding edge computing node can be embodied as a type of device, apparatus, computer, or other "thing" capable of communicating with other edge components, networking components, or endpoint components. For example, an edge computing device can be embodied as a personal computer, server, smartphone, mobile computing device, smart device, in-vehicle computing system (e.g., navigation system), a self-contained device with an enclosure, shell, etc., or other device or system capable of performing the described functions.

[0101] exist Figure 8A This is a block diagram illustrating an example implementation of an example edge computing node 800, which includes a computing engine (also referred to herein as a "computing circuit system") 802, an input / output (I / O) subsystem 808, a data storage 810, a communication circuit system subsystem 812, and optionally includes one or more peripheral devices 814. In other examples, the corresponding computing device may include other or additional components, such as those commonly found in computers (e.g., displays, peripheral devices, etc.). Additionally, in some examples, one or more of the illustrative components may be incorporated into another component or otherwise form part of another component. Figure 8A Example edge computing node 800 can be deployed in Figures 1-5 and / or Figure 7 In one of the edge computing systems illustrated in the figure, to achieve... Figures 1-5 and / or Figure 7 Any edge computing node.

[0102] Compute node 800 can be embodied as any type of engine, device, or collection of devices capable of performing various computing functions. In some examples, compute node 800 can be embodied as a single device, such as an integrated circuit, embedded system, field-programmable gate array (FPGA), system-on-a-chip (SoC), or other integrated system or device. In illustrative examples, compute node 800 includes or is embodied as processor 804 and memory 806. Processor 804 can be embodied as any type of processor capable of performing the functions described herein (e.g., executing applications). For example, processor 804 can be embodied as a multi-core processor(s), microcontroller, processing unit, dedicated or special-purpose processing unit, or other processor or processing / control circuitry.

[0103] In some examples, processor 804 may be embodied as, including or coupled to an FPGA, an application-specific integrated circuit (ASIC), reconfigurable hardware or hardware circuitry system, or other dedicated hardware for facilitating the execution of the functions described herein. Furthermore, in some examples, processor 804 may be embodied as a dedicated x-processing unit (xPU), also referred to as a data processing unit (DPU), infrastructure processing unit (IPU), or network processing unit (NPU). Such xPUs may be embodied as stand-alone circuitry or circuit packages, integrated within a SoC, or integrated with networking circuitry systems (e.g., in a smart NIC), acceleration circuitry systems, storage devices, or AI hardware (e.g., a GPU or a programmable FPGA). Beyond CPUs or general-purpose processing hardware, such xPUs may be designed to receive programming to process one or more data streams and perform specific tasks and actions for those data streams (such as mastering microservices, performing service management or orchestration, organizing or managing server or data center hardware, managing service meshes, or collecting and distributing telemetry). However, it will be understood that other variants of the xPU, SOC, CPU, and processor 804 can work together to perform various types of operations and instructions within and on behalf of the compute node 800.

[0104] Memory 806 can be embodied as any type of volatile (e.g., dynamic random access memory (DRAM), etc.) or non-volatile memory or data storage capable of performing the functions described herein. Volatile memory can be a storage medium that requires power to maintain the state of data stored therein. Non-limiting examples of volatile memory can include various types of random access memory (RAM), such as DRAM or static random access memory (SRAM). One particular type of DRAM that can be used in a memory module is synchronous dynamic random access memory (SDRAM).

[0105] In one example, the memory device is a block-addressable memory device, such as those based on NAND or NOR technology. The memory device may also include a three-dimensional crosspoint memory device (e.g., Intel® 3D XPoint™ memory) or other byte-addressable, in-place write-only non-volatile memory devices. The memory device may refer to the die itself and / or the packaged memory product. In some examples, 3D crosspoint memory (e.g., Intel® 3DXPoint™ memory) may include a transistorless, stackable crosspoint architecture where memory cells are located at the intersection of word lines and bit lines and are individually addressable, and where bit storage is based on variations in body resistance. In some examples, all or part of the main memory 806 may be integrated into the processor 804. The main memory 806 may store various software and data used during operation, such as data manipulated by one or more applications, libraries, and drivers.

[0106] The computing circuitry system 802 is communicatively coupled to other components of the computing node 800 via an I / O subsystem 808, which may be embodied as a circuitry and / or component for facilitating input / output operations with the computing circuitry system 802 (e.g., with the processor 804 and / or main memory 806) and other components of the computing circuitry system 802. For example, the I / O subsystem 808 may be embodied as or otherwise include a memory controller hub, an input / output control hub, an integrated sensor hub, firmware devices, communication links (i.e., point-to-point links, bus links, lines, cables, light guides, printed circuit board traces, etc.) and / or other components and subsystems for facilitating input / output operations. In some examples, the I / O subsystem 808 may form part of a system-on-a-chip (SoC) and may be incorporated into the computing circuitry system 802 along with one or more of the processor 804, main memory 806, and other components.

[0107] One or more descriptive data storage devices 810 may be embodied as any type of device configured for short-term or long-term data storage, such as, for example, memory devices and circuitry, memory cards, hard disk drives, solid-state drives, or other data storage devices. Each data storage device 810 may include a system partition storing data from the data storage device 810 as well as firmware code. Each data storage device 810 may also include one or more operating system partitions, depending on the type of compute node 800, to store operating system data files and executable files.

[0108] The communication circuit system 812 can be embodied as any communication circuit, device, or combination thereof capable of enabling communication between the computing circuit system 712 and other computing devices (e.g., an edge gateway of an edge computing system) via a network. The communication circuit system 812 can be configured to implement such communication using any one or more communication technologies (e.g., wired or wireless communication) and associated protocols (e.g., cellular networking protocols such as 3GPP 4G or 5G standards, wireless local area network protocols such as IEEE 802.11 / Wi-Fi®, wireless wide area network protocols, Ethernet, Bluetooth®, Bluetooth Low Energy, IoT protocols such as IEEE 802.15.4 or ZigBee®, low-power wide area network (LPWAN) or low-power wide area network (LPWA) protocols, etc.).

[0109] The illustrative communication circuitry system 812 includes a network interface controller (NIC) 820, which may also be referred to as a host infrastructure interface (HFI). The structure connected to the HFI can include various structures such as those for satellite backhaul connectivity. The NIC 820 can be embodied as one or more plug-in boards, daughter cards, network interface cards, controller chips, chipsets, or other devices that can be used by the computing node 800 to connect to another computing device, such as an edge gateway node. In some examples, the NIC 820 can be embodied as part of a system-on-a-chip (SoC) including one or more processors, or the NIC 820 can be included in a multi-chip package that also includes one or more processors. In some examples, the NIC 820 may include a local processor (not shown) and / or local memory (not shown), both of which are local to the NIC 820. In such examples, the local processor of the NIC 820 may be able to perform one or more functions of the computing circuitry system 802 described herein. Additionally or alternatively, in such examples, the NIC 820’s local memory may be integrated into one or more components of the client compute node at the board level, socket level, chip level and / or other levels.

[0110] Additionally, in some examples, the corresponding compute node 800 may include one or more peripheral devices 814. Depending on the specific type of compute node 800, such peripheral devices 814 may include any type of peripheral device typically found in compute devices or servers, such as audio input devices, displays, other input / output devices, interface devices, and / or other peripheral devices. In further examples, compute node 800 may be embodied by a corresponding edge compute node (whether a client, gateway, or aggregation node) in an edge computing system, or by a similar type of device, computer, subsystem, circuit system, or other component.

[0111] In a more detailed example, Figure 8B The figure shows a block diagram of an example edge computing node 850, which is configured to perform... Figures 16-17 Instructions for implementing the techniques described herein (e.g., operations, processes, methods, and methodologies), such as those referenced below. Figures 10-14 The described orchestrator. The edge computing node 850 provides a closer view of the corresponding components of node 800 when implemented as a computing device (e.g., a mobile device, base station, server, gateway, etc.) or a part of such a computing device. Edge computing node 850 may include any combination of hardware or logic components referenced herein, and edge computing node 850 may include any device that can be used with or coupled to an edge communication network or a combination of such networks. These components may be implemented as ICs, parts of ICs, discrete electronics or other modules, instruction sets, programmable logic or algorithms, hardware, hardware accelerators, software, firmware, or combinations thereof adapted in edge computing node 850, or implemented as components otherwise incorporated into the rack of a larger system. For example, the edge computing node 850 can be a server, personal computer, workstation, self-learning machine (e.g., neural network), mobile device (e.g., cellular phone, smartphone, tablet such as iPad™), personal digital assistant (PDA), Internet device, DVD player, CD player, digital video recorder, Blu-ray player, game console, personal video recorder, set-top box, headset or other wearable device, or any other type of computing device.

[0112] Edge computing device 850 may include a processing circuitry system in the form of a processor 852, which may be a microprocessor, multi-core processor, multi-threaded processor, ultra-low voltage processor, embedded processor, xPU / DPU / IPU / NPU, dedicated processing unit, special-purpose processing unit, or other known processing element. Processor 852 may be part of a system-on-a-chip (SoC), in which processor 852 and other components are formed as a single integrated circuit or a single package, such as an Edison™ or Galileo™ SoC board from Intel Corporation, Santa Clara, California. As an example, processor 852 may include an Intel® architecture Core™ CPU processor (such as Quark™, ​​Atom™, i3, i5, i7, i9, or MCU-class processors), or another such processor available from Intel®. However, any number of other processors may be used, such as processors available from Advanced Micro Devices (AMD®) of Sunnyvale, California; MIPS®-based designs from MIPS Technologies of Sunnyvale, California; ARM®-based designs licensed from ARM Holdings, Inc.; or processors available from customers, licensees, or adopters of the aforementioned companies. Processors may include units such as A5-A13 processors from Apple® Corporation and Snapdragon processors from Qualcomm® Technologies. TM (Snapdragon) TM The processor is either from Texas Instruments' OMAP. TM Processor. The processor 852 and its accompanying circuitry may be provided in single-socket form factor, multi-socket form factor, or various other formats, including limited hardware configurations or including fewer than Figure 8B The configuration of all components shown is illustrated below. In this example, the processor 852 implements the configuration described in the reference below. Figure 10 The example orchestrator 1010 is described.

[0113] Processor 852 can communicate with system memory 854 via interconnect 856 (e.g., a bus). Any number of memory devices can be used to provide a fixed amount of system memory. As an example, the memory can be random access memory (RAM) designed according to the Joint Electronic Devices Engineering Committee (JEDEC), such as DDR or mobile DDR standards (e.g., LPDDR, LPDDR2, LPDDR3, or LPDDR4). In specific examples, memory components can conform to standards promulgated by JEDEC, such as JESD79F for DDR SDRAM, JESD79-2F for DDR2 SDRAM, JESD79-3F for DDR3 SDRAM, JESD79-4A for DDR4 SDRAM, JESD209 for low-power DDR (LPDDR), JESD209-2 for LPDDR2, JESD209-3 for LPDDR3, and JESD209-4 for LPDDR4. Such standards (and similar standards) can be referred to as DDR-based standards, and the communication interface of the storage device implementing such standards can be referred to as a DDR-based interface. In various implementations, individual memory devices can be any number of different package types, such as single-die package (SDP), dual-die package (DDP), or quad-die package (Q17P). In some examples, these devices can be directly soldered to the motherboard to provide a low-profile solution, while in other examples, the devices are configured as one or more memory modules that are then coupled to the motherboard via a given connector. Any number of other memory implementations can be used, such as other types of memory modules, for example, different kinds of dual in-line memory modules (DIMMs), including but not limited to microDIMMs or miniDIMMs.

[0114] To provide persistent storage for information such as data, applications, operating systems, etc., storage 858 may also be coupled to processor 852 via interconnect 856. In this example, storage 858 may be implemented via a solid-state drive (SSDD). Other devices that can be used for storage 858 include flash memory cards (such as SD cards, microSD cards, XD picture cards, etc.) and USB flash drives. In the example, the memory device may be or may include a memory device using chalcogenide glass, multi-threshold level NAND flash memory, NOR flash memory, single-level or multi-level phase change memory (PCM), resistive memory, nanowire memory, ferroelectric transistor random access memory (FeTRAM), antiferroelectric memory, magnetoresistive random access memory (MRAM) incorporating memristor technology, resistive memory including metal oxide substrate, oxygen vacancy substrate and conductive bridge random access memory (CB-RAM), or spin-transfer torque (STT)-MRAM, a device based on spintronic magnetic junction memory, a device based on magnetic tunneling junction (MTJ), a device based on DW (domain wall) and SOT (spin-orbit transfer), a thyristor-based memory device, or any combination of the above or other memories.

[0115] In a low-power implementation, memory 858 may be an on-die persistent memory or register associated with processor 852. However, in some examples, memory 858 may be implemented using a micro hard disk drive (HDD). Furthermore, any number of new technologies, such as resistive random access memory, phase-change memory, holographic memory, or chemical memory, may be used for memory 858, in addition to or in lieu of the described technologies.

[0116] Components can communicate via Interconnect 856. Interconnect 856 can include any number of technologies, including Industry Standard Architecture (ISA), Extended ISA (EISA), Peripheral Component Interconnect (PCI), Peripheral Component Interconnect Extended (PCIx), PCI Express (PCI Fast, PCIe), or any other technologies. Interconnect 856 can be, for example, a proprietary bus used in a SoC-based system. Other bus systems can be included, such as I2C interfaces, SPI interfaces, point-to-point interfaces, power buses, and so on.

[0117] Interconnect 856 couples processor 852 to transceiver 866 for, for example, communication with a connected edge device 862. Transceiver 866 can use any number of frequencies and protocols, such as 2.4 GHz transmission under the IEEE 802.15.4 standard, using standards such as Bluetooth® Low Energy (BLE) defined by the Bluetooth® Special Interest Group, or the ZigBee® standard, etc. Any number of radios configured for a specific wireless communication protocol can be used for connectivity to the connected edge device 862. For example, a Wireless Local Area Network (WLAN) unit can be used to implement Wi-Fi® communication according to the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard. Additionally, Wireless Wide Area Communication (WWAN) can occur via the WWAN unit, for example, according to cellular or other wireless wide area protocols.

[0118] Wireless network transceivers 866 (or multiple transceivers) can communicate using various standards or radios for communication at different ranges. For example, an edge computing node 850 can use a local transceiver based on BLE or another low-power radio to communicate with nearby devices (e.g., within approximately 10 meters) to save power. Edge devices 862 connected at greater distances (e.g., within approximately 50 meters) can be contacted via ZigBee® or other intermediate-power radios. These two communication technologies can occur at different power levels via a single radio, or via separate transceivers, such as a local transceiver using BLE and separate mesh transceivers using ZigBee®.

[0119] A wireless transceiver 866 (e.g., a radio transceiver) may be included to communicate with devices or services in the edge cloud 890 via LAN or WAN protocols. The wireless transceiver 866 may be an LPWA transceiver conforming to standards such as IEEE 802.15.4 or IEEE 802.15.4g. The edge computing node 850 may communicate over a wide area using LoRaWAN™ (Long Range Wide Area Network) developed by Semtech and the LoRa Alliance. The technologies described herein are not limited to these technologies but can be used with any number of other cloud transceivers that enable long-range, low-bandwidth communication, such as Sigfox and other technologies. Furthermore, other communication technologies, such as time-division channel hopping as described in the IEEE 802.15.4e specification, may be used.

[0120] In addition to the systems mentioned for the wireless network transceiver 866 described herein, any number of other radio communications and protocols may be used. For example, transceiver 866 may include a cellular transceiver that uses spread spectrum (SPA / SAS) communication to enable high-speed communication. Furthermore, any number of other protocols may be used, such as Wi-Fi® networks for medium-speed communication and provisioning network communication. Transceiver 866 may include radios compatible with any number of 3GPP (3rd Generation Partnership Project) specifications, such as Long Term Evolution (LTE) and 5th Generation (5G) communication systems, which are discussed in further detail at the end of this disclosure. Network interface controller (NIC) 868 may be included to provide wired communication to nodes of edge cloud 890 or to other devices, such as edge devices 862 connected (e.g., operating in a mesh). Wired communication may provide Ethernet connectivity or may be based on other types of networks, such as Controller Area Network (CAN), Local Interconnect Network (LIN), DeviceNet, ControlNet, Data Highway+, Fieldbus (PROFIBUS), or Industrial Ethernet (PROFINET), etc. An additional NIC 868 may be included to enable connectivity to a second network; for example, the first NIC 868 provides communication to the cloud via Ethernet, and the second NIC 868 provides communication to other devices via another type of network. The NIC 868 may include instructions 882 in the form of software, firmware, or hardware commands for implementing the techniques described herein.

[0121] Given the diversity of applicable communication types from a device to another component or network, the applicable communication circuitry used by the device may include, or be embodied by, any one or more of components 864, 866, 868, or 870. Therefore, in the various examples, applicable means for communication (e.g., receiving, transmitting, etc.) may be embodied by such communication circuitry.

[0122] Edge computing node 850 may include or be coupled to acceleration circuitry system 864, which may be embodied by one or more AI accelerators, neural compute sticks, neuromorphic hardware, FPGAs, GPU deployments, xPU / DPU / IPU / NPU deployments, one or more SoCs, one or more CPUs, one or more digital signal processors, application-specific ASICs, or other forms of dedicated processors or circuitry systems designed to perform one or more proprietary tasks. These tasks may include AI processing (including machine learning, training, inference, and classification operations), visual data processing, network data processing, object detection, rule analysis, etc. These tasks may also include specific edge computing tasks for service management and service operations discussed elsewhere in this document.

[0123] Interconnect 856 can couple processor 852 to a sensor hub or external interface 870 for connecting additional devices or subsystems. Devices may include sensors 872, such as accelerometers, level sensors, flow sensors, optical sensors, camera sensors, temperature sensors, global positioning system (e.g., GPS) sensors, pressure sensors, barometric pressure sensors, and so on. The hub or interface 870 may further be used to connect edge computing node 850 to actuators 874, such as power switches, valve actuators, audible sound generators, visual warning devices, and so on.

[0124] In some optional examples, various input / output (I / O) devices may be present within or connected to the edge computing node 850. For example, a display or other output device 884 may be included to display information such as sensor readings or actuator positions. Input devices 886 (such as touchscreens or keypads) may be included to accept input. Output devices 884 may include any number of audio or visual display forms, including: simple visual outputs such as binary status indicators (e.g., LEDs); multi-character visual outputs; or more complex outputs such as displays (e.g., LCD screens) with outputs of characters, graphics, multimedia objects, etc., generated or produced from the operation of the edge computing node 850. In the context of this system, the display or console hardware may: be used to provide outputs and receive inputs from the edge computing system; be used to manage components or services of the edge computing system; identify the status of edge computing components or services; or be used for any other number of management or administrative function or service use cases.

[0125] Battery 876 can power edge computing node 850, but in an example where edge computing node 850 is mounted in a fixed location, edge computing node 850 may have a power source coupled to the power grid, or the battery may be used as a backup or for temporary functions. Battery 876 can be a lithium-ion battery, a metal-air battery (such as a zinc-air battery, an aluminum-air battery, a lithium-air battery), etc.

[0126] A battery monitor / charger 878 may be included in an edge computing node 850 to track the state of charge (SoCh) of a battery 876 (if included). The battery monitor / charger 878 may be used to monitor other parameters of the battery 876 to provide fault prediction, such as the state of health (SoH) and state of function (SoF) of the battery 876. The battery monitor / charger 878 may include a battery monitoring integrated circuit, such as the LTC4020 or LTC2990 from Linear Technologies, the ADT7488A from ON Semiconductor in Phoenix, Arizona, or an IC from the UCD90xxx family from Texas Instruments in Dallas, Texas. The battery monitor / charger 878 may pass information about the battery 876 to a processor 852 via interconnect 856. The battery monitor / charger 878 may also include an analog-to-digital converter (ADC) that enables the processor 852 to directly monitor the voltage of the battery 876 or the current from the battery 876. Battery parameters can be used to determine the actions that the edge computing node 850 can perform, such as transmission frequency, mesh network operation, sensing frequency, and so on.

[0127] Power block 880 or other power sources coupled to the grid can be coupled to battery monitor / charger 878 to charge battery 876. In some examples, power block 880 can be replaced by a wireless power receiver to obtain power wirelessly, for example, via a loop antenna in edge computing node 850. Wireless battery charging circuitry, such as the LTC4020 chip from Linear Technologies, Inc., Milpitas, California, etc., can be included in battery monitor / charger 878. Specific charging circuitry can be selected based on the size of battery 876 and therefore based on the required current. Charging can be performed using the Airfuel standard issued by the Airfuel Alliance, the Qi wireless charging standard issued by the Wireless Power Consortium, or the Rezence charging standard issued by the Alliance for Wireless Power, etc.

[0128] Storage 858 may include instructions 882 in the form of software, firmware, or hardware commands for implementing the techniques disclosed herein. Although such instructions 882 are shown as blocks of code included in memory 854 and storage 858, it will be understood that any of the blocks of code may be replaced by hard-wired circuitry, for example, built into an application-specific integrated circuit (ASIC).

[0129] In the example, instructions 882 provided via memory 854, storage 858, or processor 852 may be embodied in a non-transient machine-readable medium 860, which includes code for instructing processor 852 to perform electronic operations in edge computing node 850. Processor 852 can access non-transient machine-readable medium 860 via interconnect 856. For example, non-transient machine-readable medium 860 may be embodied by a device described for storage 858, or may include specific storage units such as optical discs, flash drives, or any number of other hardware devices. Non-transient machine-readable medium 860 may include instructions for instructing processor 852 to perform specific sequences or flows of actions, such as those described with reference to the flowcharts and block diagrams(s) of operations and functions depicted above and below. As applicable herein, the terms "machine-readable medium" and "computer-readable medium" are interchangeable.

[0130] Furthermore, in a specific example, instructions 882 on processor 852 (either alone or in combination with instructions 882 on machine-readable medium 860) can configure the execution or operation of a Trusted Execution Environment (TEE) 890. In the example, TEE 890 operates as a protected region accessible to processor 852 for secure execution of instructions and secure access to data. Various implementations of TEE 890, along with the accompanying secure region in processor 852 or memory 854, can be provided, for example, using Intel® Software Protection Extensions (SGX) or ARM® TrustZone® Hardware Security Extensions, Intel® Management Engine (ME) or Intel® Converged Security Manageability Engine (CSME). Security hardening, hardware root of trust, and other aspects of trusted or protected operation can be implemented in device 850 via TEE 890 and processor 852.

[0131] In further examples, machine-readable media also includes any tangible medium capable of storing, encoding, or carrying instructions for execution by a machine and causing the machine to perform any one or more methods of this disclosure, or a tangible medium capable of storing, encoding, or carrying data structures utilized by or associated with such instructions. "Machine-readable media" can therefore include, but is not limited to, solid-state memory, optical media, and magnetic media. Specific examples of machine-readable media include non-volatile memory, including, but not limited to: semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices); magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Instructions embodied in the machine-readable medium can be transmitted or received via a communication network using a transmission medium, via a network interface device, utilizing any of a variety of transmission protocols (e.g., HTTP).

[0132] Machine-readable media can be provided by storage devices or other means capable of controlling data in a non-transient format. In the example, information stored on or otherwise provided on a machine-readable medium can represent instructions, such as the instructions themselves or a format from which instructions can be derived. Such a format from which instructions can be derived can include source code, encoded instructions (e.g., in compressed or encrypted form), packaged instructions (e.g., divided into multiple packages), etc. Information representing instructions in a machine-readable medium can be processed by a processing circuitry system into instructions to implement any of the operations discussed herein. For example, deriving instructions from information (e.g., processing performed by processing circuitry) can include: (e.g., from source code, object code, etc.) compiling, interpreting, loading, organizing (e.g., linking dynamically or statically), encoding, decoding, encrypting, decrypting, packing, unpacking, or otherwise manipulating information into instructions.

[0133] In the example, instruction derivation may include (e.g., by processing a circuit system) assembling, compiling, or interpreting information to create instructions from some intermediate or preprocessed format provided by a machine-readable medium. When information is provided in multiple parts, it can be combined, unpacked, and modified to create instructions. For example, information may reside in multiple compressed source code packages (or object code, or binary executable code, etc.) on one or more remote servers. The source code packages may be encrypted during transmission over a network and may be decrypted, decompressed, assembled (e.g., linked), and compiled or interpreted (e.g., compiled or interpreted into libraries, standalone executables, etc.) at the local machine and executed by the local machine.

[0134] Figures 16-17The machine-executable instructions 1600 may be stored in mass storage device 828, in volatile memory 814, in non-volatile memory 816, and / or on a removable non-transient computer-readable storage medium such as a CD or DVD.

[0135] Therefore, edge computing utilizes local edge devices or nodes located in parts of the network to process data remotely from centralized clouds or solely based on cloud storage. Edge computing provides distributed storage, and local nodes or devices can act as servers within the edge computing infrastructure for processing data, providing services, etc. Alternatively or additionally, fog computing combines centralized data storage with cloud-based storage. For example, processing in a fog computing network occurs on a local network with distributed servers, allowing data to be accessed offline because at least some portion of the data is stored locally, as opposed to cloud computing where computation and storage are performed using remote servers. In the case of fog computing, because portions of the data can be stored and processed locally, unstable network connections and / or other network downtime do not interfere with operation.

[0136] Using one or more example environments described above, services can be orchestrated by one or more orchestrators within and between multiple edge and / or cloud resources. Services such as augmented reality, virtual reality, and content delivery networks (CDNs) can be executed by one or more edge, cloud, or fog devices under the direction of one or more orchestrators.

[0137] Figure 9 An example edge architecture 900 is shown, which includes multiple layers 910 with associated network latency 920 and limitations 930. (As shown in...) Figure 9 As shown in the example, different services are executed in different locations within the hierarchy of the example edge architecture 900 (e.g., similar to or different from the example edge environment 110, edge cloud 210, etc. described above).

[0138] For example, in a small cell layer 911 with a round-trip network latency of less than 1 millisecond, the available computing power is less than 500 watts (W); the form factor is small box; the heat is ambient; there is no physical monitoring for security; and management is remote. Example use cases for small cell layer 911 include smart cities, vehicle-to-vehicle (V2V) communication, retail, video analytics, and more. Key performance indicators (KPIs) associated with these example use cases include data privacy, backhaul bandwidth savings, reliability, and latency.

[0139] As another example, consider an on-premises setup and / or small cloud 913 with a round-trip network latency of less than 1 ms 920, where the available computing power is approximately 10 kilowatts (kW); the form factor is one or more computing and / or communication equipment racks; thermal compliance with Network Equipment Building Systems (NEBS) or standard data center (DC) thermal guidelines; physical monitoring for security may or may not be present; and management is remote. Use cases for this example on-premises setup / small cloud 913 include AR / VR, retail, real-time (RT) streaming, healthcare, etc. KPIs associated with these example use cases include latency, backhaul traffic savings, network scalability, data privacy, reliability, and access to services.

[0140] In example cell tower 915 with a round-trip network latency of 1-5 ms (920), available computing power is less than 500 W; the form factor is a pizza box; heat is measured according to the NEBS Environmental Guidelines; there is no physical monitoring for security; and management is remote. Use cases for example cell tower 915 include smart cities, V2V, video analytics, drones / Internet of Things (IoT), rural areas, etc. KPIs associated with these example use cases include data privacy, backhaul traffic savings, reliability, throughput, latency, and access to services.

[0141] In an example layer 917 with multiple aggregation points and a round-trip network latency of 5 ms plus 1-2 ms (per 100 km) 920, the available computing power is 9 kW per rack and 1 kW per square meter (sqm) for the multipoint array 917; the form factor is multiple computing and / or communication equipment racks; heat is based on NEBS and / or standard DC conditions; physical monitoring for security may or may not be present; and management is remote. Use cases for the example multipoint layer 917 include smart cities, video analytics, drones / IoT, healthcare, CDN and storage, FaaS, AR / VR / Mixed Reality (MR), etc. KPIs associated with these example use cases include data privacy, reliability, backhaul traffic savings, throughput, latency, etc.

[0142] In the example core layer 919 with a round-trip network latency of 5 ms plus 5 ms (per 400km) 920, available computing power, CDN formation, storage gateway (GW), etc. KPIs associated with these example use cases include backhaul traffic savings, throughput, etc.

[0143] For example, after the example core layer 919 is the cloud, where round-trip network latency may be more than 60 ms.

[0144] In some examples, each edge location (such as, Figure 9Each edge layer 910 in the example includes a local orchestrator (not shown, also referred to as an auxiliary orchestrator, such as example orchestrators 142, 150, example orchestrator 560, example container orchestrator 631, etc.) that manages the edge layer 910 from the device to the data center edge. An edge orchestrator (not shown, also referred to as a master orchestrator) coordinates among the local orchestrators to manage services across multiple edge layers 911-919. Thus, the edge orchestrator provides edge-to-edge (e2e) orchestration between local orchestrators.

[0145] For example, an edge orchestrator can instantiate and allocate resources across selected edge locations to perform services within specific SLAs and at specific costs. For instance, an edge orchestrator can monitor the SLAs associated with an application during its execution and compare the monitored SLAs to the provided SLAs. An edge orchestrator can migrate services from one edge location to another based on one or more criteria and / or factors, such as additional resources to satisfy the SLA, adaptation to user movement for continued access to the service (e.g., a user in a car moving from one location to another).

[0146] However, different edge layers or locations 910 can be defined, owned, and / or controlled by different edge entities, resulting in differences in management, authorization, monitoring, etc. Edge orchestrators can cross edge boundaries and overcome, resolve, and / or otherwise react to these edge-specific issues that constrain local orchestrators.

[0147] Figure 10 This is a block diagram of an example orchestration system 1000, which includes a master orchestrator 1010 (also referred to as an edge orchestrator 1010), multiple auxiliary orchestrators 1020-1024, and multiple edge devices or other edge entities 1030-1034. The auxiliary orchestrators 1020-1024 are edge entities delegated to each edge entity 1030-1034 and execute under the master orchestrator 1010, which acts as the primary orchestration mechanism. Thus, the auxiliary orchestrators 1020-1024 can form a mesh, and the master orchestrator 1010 coordinates or orchestrates the mesh of the orchestrators 1020-1024. Orchestrators 1010, 1020-1024 can be implemented as one or more containers, virtual machines, other hardware and / or software constructs, processors, etc., such as example orchestrators 142, 150, example orchestrator 560, example container orchestrator 631, etc.

[0148] The delegated orchestrator enables improved service management between edge devices 1030-1034. Delegated orchestration, used between the primary orchestrator 1010 and the secondary orchestrators 1020-1024, controls the scheduling and management of services (and / or associated tasks, jobs, etc.) throughout the service's lifecycle, regardless of whether all or part of the service is relayed, partitioned, or migrated between edge entities 1030-1034.

[0149] For example, a master orchestrator 1010 may delegate services from a first edge device among a plurality of edge devices 1030-1034 to a second edge device among the plurality of edge devices 1030-1034, and associated auxiliary orchestrators 1020-1024 are assumed to be responsible for orchestrating services on edge devices 1030-1034. Thus, some examples provide delegated orchestration to orchestrate multiple auxiliary orchestrators 1020-1024 within the scope of the master orchestrator 1010, rather than providing a peer-to-peer model that attempts to negotiate separate edge deployments belonging to different entities and being orchestrated independently. In some examples, the delegated orchestration authority is provided based on the verification and validation of the credibility of the delegated authority.

[0150] Figure 11 The diagram illustrates an example network environment or infrastructure 1100 including a master orchestrator 1010 and auxiliary orchestrators 1020-1022. (Example:) Figure 11 As shown in the example, the master orchestrator 1010 communicates with the end-to-end service logic 1110. The end-to-end service logic 1110 provides differentiated subscriber SLOs and continuous service to one or more subscribers or tenants in the network environment or infrastructure 1100. In some examples, the end-to-end service logic 1110 provides service, hardware, and network telemetry to one or more subscribers. For example, the edge orchestrator 1010 can reconfigure service, hardware, and / or network telemetry.

[0151] like Figure 11 As shown in the example, the master orchestrator 1010 can communicate with the hardware and / or software resources of edge entities 1030 and 1032 via auxiliary orchestrators 1020 and 1022. The master orchestrator 1010 allows the merging of multiple services for multiple tenants to provide dynamic adaptive SLA / SLO guarantees with improved security. The combination of the master orchestrator 1010 and the auxiliary orchestrators 1020-1022 enables self-management and orchestration across hardware and software resources.

[0152] like Figure 11As shown in the example, the auxiliary orchestrator 1020 provides service telemetry for the configuration and control of software and / or hardware resources, such as co-located applications 1120, edge nodes 1122, network function virtualization (NFV) 1124, etc. Example applications 1120, edge nodes 1122 (e.g., example edge nodes 148, 156, 522, 524, 720, 740, 800, 850), and / or NFV 1124 can be used to interact with one or more edge software development kits (SDKs) 1126 (such as data plane development kits, deep learning and / or other artificial intelligence network model kits, etc.).

[0153] In example infrastructure 1100, the edge SDK 1126 executes on top of the Virtualization Infrastructure Manager (VIM) 1128, which manages other compute, storage, and / or network resources of NFV 1124 and / or edge entities 1030. Example VIM 1128 manages an inventory of virtual resource-to-physical resource allocations. For example, allocation management allows for the orchestration of allocation, upgrades, releases, and reclamation of NFV 1124 resources, as well as improved and optimized use of NFV 1124 resources by orchestrator 1020. The Multi-Access Edge Computing (MEC) platform 1130 enables cloud computing and services at the network edge and allows VIM 1128 to interact with shared edge infrastructure 1132. The shared edge infrastructure 1132 includes one or more servers, radios, etc., that can be used to communicate with near-edge resources 1134 and / or edge resources 1135-1136, such as cell towers, data centers, other communication resources, etc.

[0154] Alternatively or additionally, for example, edge node 1122 and / or NFV 1124 may communicate directly with edge resources 1135-1136 using Local Branch Outlet (LBO) switch 1138. Edge entity 1030 may connect to another edge entity 1032 via backhaul network 140. For example, master orchestrator 1010 may communicate with backhaul network 1140 to provide network telemetry (e.g., aggregation, per-slice, etc.) to tune network configuration. Additionally, for example, master orchestrator 1010 may exchange hardware telemetry information with shared edge infrastructure 1132 and reconfigure the hardware in shared edge infrastructure 1132.

[0155] The second edge device or entity 1032 may include elements similar to and / or different from the example edge entity 1030. For example... Figure 11As shown in the example, edge device 1032 includes NFV 1142, a control plane 1144, and one or more other tenants 1146 that interact with example orchestrator 1022. Orchestrator 1022 utilizes VIM 1148 to interact with cloud platform 1150 and shared data center 1152, which communicate with edge data center 1154.

[0156] exist Figure 11 In the example system or infrastructure 1100 shown, orchestration can be coordinated across multiple edge devices / entities 1030-1032 using a master orchestrator 1010, and orchestration can be coordinated within a specific edge domain 1030-1032 using auxiliary orchestrators 1020-1022. In such a system 1100, orchestration responsibilities and / or authority are delegated from auxiliary orchestrator 1020 to auxiliary orchestrator 1022 to help ensure that services can be performed across edge boundaries within the example infrastructure 1100.

[0157] Figure 12 The diagram illustrates another example network architecture 1200 for delegation used to implement orchestration between auxiliary orchestrators 1020-1022. (See diagram for example.) Figure 12 As shown in the example, the auxiliary orchestrator 1020 provides service telemetry for configuring and controlling software and / or hardware resources, such as co-located application 1120, edge node 1122, NFV 1124, etc. Example application 1120, edge node 1122, and / or NFV 1124 can be used to interact with one or more edge SDKs 1126, such as Data Plane Development Kit (DPDK), deep learning and / or other artificial intelligence network model toolkits, etc.

[0158] In example infrastructure 1200, edge SDK 1126 executes on top of VIM 1128, which manages NFV 1124 and / or other compute, storage, and / or network resources. Example VIM 1128 manages an inventory of virtual-to-physical resource allocations. For example, allocation management allows for the orchestration of allocation, upgrades, releases, and reclamation of NFV 1124 resources, as well as the improvement and optimization of NFV 1124 resource usage by orchestrator 1020. MEC platform 1130 implements cloud computing and services at the network edge and allows VIM 1128 to interact with shared edge infrastructure 1132, which includes one or more servers, radios, etc. Alternatively or additionally, edge nodes 1122 and / or NFV 1124 may communicate directly with edge infrastructure 1132 and / or other edge resources using LBO switch 1138.

[0159] exist Figure 12 In the example, orchestrator 1022 includes an orchestration stack that includes a validator 1210, a delegation manager 1212, one or more interfaces 1214, and one or more orchestration delegations 1216-1218.

[0160] like Figure 12 As shown in the example, the auxiliary orchestrator 1020 provides service telemetry for configuring and controlling software and / or hardware resources, such as co-located application 1220, edge node 1222, NFV 1224, etc. The example application 1220, edge node 1222, and / or NFV 1224 can be used to interact with one or more edge SDKs 1226, such as DPDK, deep learning, and / or other artificial intelligence network model toolkits.

[0161] In example infrastructure 1200, edge SDK 1226 executes on top of VIM 1228, which manages NFV 1124 and / or other compute, storage, and / or network resources. Example VIM 1228 manages an inventory of virtual-to-physical resource allocations. For example, allocation management allows for the orchestration of allocation, upgrades, releases, and reclamation of NFV 1224 resources, as well as the improvement and optimization of NFV 1124 resource usage by orchestrator 1022. MEC platform 1230 implements cloud computing and services at the network edge and allows VIM 1228 to interact with shared edge infrastructure 1232, which includes one or more servers, radios, etc. Alternatively or additionally, for example, edge nodes 1222 and / or NFV 1224 can communicate directly with edge infrastructure 1232 and / or other edge resources using LBO switch 1238.

[0162] Return to Figure 12 The example orchestrator 1022's orchestration stack, each orchestration delegation 1216-1218 can be implemented as a plug-in or orchestration instance representing an edge entity or agency A 1032 instantiated within the orchestration stack that runs and manages edge locations owned and managed by entity B 1030. The example verifier 1210 can be used to verify and / or otherwise validate files (e.g., binaries and / or other executables) sent from another orchestrator 1010, 1020, and / or other sources used for execution on the edge using orchestrator 1022. The example interface 1214 enables the execution of SLA / SLO / other requests, exceptions, and / or requirements.

[0163] For example, one or more interfaces 1214 can be used to specify the services to be executed in conjunction with metadata (e.g., files, knobs, etc.), SLAs (e.g., required resource amounts, etc.), SLOs (frames per second, other application KPIs, etc.), maximum costs, etc. The orchestrator 1022 can receive the services to be executed via interface 1214 and can expose interface 1214 to periodically check the status of one or more executing services. Additionally, interface 1214 can be used to retrieve historical information to evaluate performance, system behavior, etc., over time.

[0164] The example orchestration manager 1212 manages one or more orchestration delegates 1216-1218 to allocate services for deployment on edge entity 1030. In some examples, the edge is owned by entity B 1030, and orchestration delegates 1216-1218 can manage a subset of resources of edge entity 1030 within a deployment. For example, entity B allows orchestration delegates 1216-1218 to have control over two nodes 1222 of edge location 1030. Subsequently, orchestration delegates 1216-1218 can select resources and orchestrate the deployment of services on those resources without working with the orchestrator 1020 of entity B 1030.

[0165] Alternatively or additionally, orchestration delegates 1216-1218 perform handshakes with orchestrator 1020 of entity B 1030 to manage the edge and coordinate resource levels, thereby meeting SLA / SLO / other requirements while attempting to minimize or otherwise reduce costs. Orchestration delegates 1216-1218 may obtain telemetry from orchestrator 1020 on demand, periodically, and / or based on triggers, etc. Telemetry information can be used by orchestration delegates 1216-1218 to predict future resource usage and identify the possibility of reduced costs due to lower loads. Thus, for example, orchestration delegates 1216-1218 monitor (multiple) services and execute one or more closed-loop control loops to help ensure that SLOs, SLAs, etc., are met for that service(s). Orchestration delegates 1216-1218 may perform handshakes between edge entity 1030 and its orchestrator 1020 to allocate additional resources when SLOs, SLAs, etc., are not met. In some examples, when SLOs, SLAs, etc., cannot be satisfied by the current edge configuration, orchestration delegates 1216-1218 contact the master orchestrator 1010 to trigger service migration.

[0166] In some examples, orchestration delegates 1216-1218 are "onboard" or configured for specific edge locations (e.g., owned by entity B 1030, etc.). To install orchestration delegates 1216-1218 to orchestrate edge entity B 1030, an auxiliary orchestrator 1022 of orchestration entity A 1032 includes orchestration modules 1216-1218 specifically for edge deployment of delegations to entity B 1030. Originating entity A provides delegation 1216-1218 as a binary file along with a signature of that binary file. Verifier 1210 communicates with a verification server (not shown, e.g., owned by a trusted entity such as a verification authority, etc.) to verify the binary file. For example, verifier 1210 may calculate a hash of the binary file and send that hash, signature, and the binary file to the verification server to verify the binary file. If the verifier 1210 determines that the orchestration delegates 1216-1218 are trustworthy, the orchestration manager 1212 of the delegate can trigger the instantiation of the orchestration delegates 1216-1218.

[0167] Thus, orchestrators 1020-1024 form a mesh of orchestrators 1020-1024, wherein each orchestrator 1020-1024 can relay services and / or associated data from one orchestrator 1020-1024 to another, based on defined SLOs, SLAs, and / or other requirements / obligations, and can share the responsibility of performing services. The master orchestrator or edge orchestrator 1010 can coordinate operations among the auxiliary orchestrators 1020-1024 and adapt to failures, defects, allocation problems, bandwidth, etc., by triggering assignments to different orchestrators 1020-1024. However, in some examples, the auxiliary orchestrators 1020-1024 communicate among themselves (e.g., via orchestration delegation 1216-1218) to migrate services, reallocate resources, share SLO / SLA responsibilities, etc., without utilizing the edge orchestrator 1010.

[0168] In some examples, orchestration delegation 1216-1218 includes a token that contains information related to the service to be performed, the parameters / constraints of the service, the associated SLO / SLA, and / or other information related to the service. The token can be updated and / or otherwise edited to indicate the progress of the service toward completion (e.g., completion of execution, completion of SLO and / or SLA, etc.). The auxiliary orchestrator 1022 can pass the token to the next orchestrator 1024 (e.g., via orchestration delegation 1216-1218, via interface 1214, via the delegated orchestration manager 1212, etc.) to inform the auxiliary orchestrator 1024 about what the auxiliary orchestrator 1022 has done and what the auxiliary orchestrator 1024 will do now. In some examples, auxiliary orchestrators 1022 and 1024 negotiate to determine which edge entity is responsible for which tasks. For example, such negotiations can occur between orchestration delegates 1216-1218 and / or the associated orchestration manager 1212.

[0169] In some examples, edge infrastructure 1000, 1100, and 1200 can be extended to include "fog" computing. As described above, fog computing involves transmitting data from a collection point to a gateway for processing. The processed data is then sent back to the edge for action. Fog computing uses edge devices and gateways with local area networks (LANs) for processing. Thus, fog can combine cloud and edge computing for data processing, service execution, and more.

[0170] Figure 13 This is a block diagram of an example fog network 1300, which includes a fog management domain (FMD) 1310 implemented in a cloud and / or at the edge 1320 (e.g., example edge cloud 210, etc.). The FMD 1310 communicates with multiple fog nodes (FNs) 1330-1336, including a fog node elected leader (FNeL) 1340. The FNeL 1340 may be determined by the FMD 1310 and / or by the FNs 1330-1336. The FNeL 1340 and / or the FMD 1310 can manage the set of FNs 1330-1336 and their associated resources.

[0171] Therefore, for contracts that perform services, one or more subcontract obligations can be delegated to (multiple) FN1330-1336 nodes for execution at the edge. For example, the execution of tasks on FN1330-1336 can be coordinated using FMD 1310, FNeL 1340, etc., in conjunction with the master orchestrator 1010 and / or auxiliary orchestrators 1020-1024. Using FN1330-1336 and / or (multiple) edge nodes 1122, 1222, etc., trusted contracts for one or more services (e.g., communication, software tracing, etc.) can be divided into functions for execution by multiple resources (such as (multiple) fog nodes 1330-1336, edge nodes 1122, 1222, etc.).

[0172] In some examples, orchestrators 1010, 1020-1024 can be used to automate the repair of problems in fog 1300, problems on the edge 1100, 1200, problems in the cloud, etc. For example, tracing software execution can be used to identify and repair problems that accompany the execution of services. For example, the allocated memory can also be increased or decreased depending on the repair workflow.

[0173] Using orchestrators 1010, 1020-1024, services can be managed, application performance can be monitored and / or tuned, and services and / or other applications can be distributed across cloud, edge, and fog resources. For example, auxiliary orchestrators 1020-1024 (e.g., and orchestration modules 1216-1218, etc., delegated by the auxiliary orchestrators) can manage execution on specific edge or fog entities 1030-1032, while the master orchestrator or edge orchestrator 1010 manages execution and resources across edge, fog, and / or cloud infrastructure. For example, multiple orchestration, resource management, security, and / or other policies can be coordinated by the master orchestrator 1010. For example, interfaces and / or APIs can be exposed for interaction between orchestrators 1020-1024.

[0174] The set of orchestrators 1020-1024 acts as a mesh of intermediaries or proxies to achieve transparency between architectures. The mesh of orchestrators 1020-1024 enables cross-platform awareness of service flows, SLOs, SLAs, etc. Each auxiliary orchestrator 1020-1024 operates on behalf of an associated edge entity 1030-1034 to execute services on entity 1030-1034, move all or part of service execution to another entity 1030-1034, etc. For example, the master orchestrator 1010 oversees the mesh of auxiliary orchestrators 1020-1024 and helps ensure communication between the meshes of orchestrators 1020-1024 to execute the service(s)(s)(s)(s)(s)(s)(s)) according to the contract / SLA / SLO(s) ...

[0175] Figure 14 This is an example network infrastructure 1400 in which multiple edge / cloud resources 1410-1412 accommodate multiple FMDs 1420-1422 communicating with FNeLs 1430-1434 to coordinate resource allocation, configuration, and / or other activities among multiple FNs 1440-1451. Thus, for example, an orchestrator may be implemented in each FMD 1420-1422 to coordinate resource allocation, configuration, and / or other activities. Alternatively or additionally, for example, auxiliary orchestrators may be implemented in each FNeL 1430-1432, wherein a master orchestrator is implemented in each FMD 1420-1422 to coordinate among these auxiliary orchestrators.

[0176] Figure 15 An example data stream 1500 is shown between orchestrator 1020, orchestrator 1022, and verification server 1501, for example, used to configure orchestration delegation(s) 1216-1218 and facilitate the execution of services and the allocation of additional resources(s) for executing services. At 1502, an orchestration delegation file (e.g., a binary file representing orchestration delegation(s) 1216-1218, etc.) is provided from orchestrator 1020 to orchestrator 1022. At 1504, orchestrator 1022 requests verification of the file from verification server 1501. Verification server 1501 can verify the file by calculating a hash, analyzing a signature, evaluating the file's source, etc. At 1506, when the file is verified, the verification is sent back to orchestrator 1022. At 1508, orchestrator 1022 instantiates orchestration delegation(s) 1216-1218 from the file.

[0177] At 1510, orchestration delegation 1216-1218 is used to orchestrate the execution of the service. Orchestration delegation 1216-1218 may include information about the edge (or fog) entities 1030-1032 on which the service is to be executed, as well as the contracts managing the execution of the service and the associated SLOs and SLAs. At 1512, orchestrator 1022 may determine that additional / alternative resources should be allocated for the execution of the service and trigger allocation to orchestrator 1020. At 1514, orchestrator 1020 is assumed to be responsible for the execution of the service. Thus, orchestrators 1020 and 1022 can collaborate to help ensure that the service is executed with appropriate resources, thereby satisfying the contracts managing the execution of the service and / or other SLOs / SLAs.

[0178] The following items are implemented by logic circuits such as hardware processors: Example Master Orchestrator 1010, Example Auxiliary Orchestrators 1020-1024, Example Edge Entities 1030-1034, Example End-to-End Service Logic 1110, Example Application 1120, Example Edge Nodes 1122, 1222, Example NFV 1124, 1142, 1224, Example Edge SDK 1126, 1226, Example VIM 1128, 1148, 1228, Example MEC 1130, 1230; Example shared edge infrastructure 1132, 1232; Example edge resources 1134-1136, 1154; Example LBO switches 1138, 1238; Example backhaul 1140; Example control plane 1144; Example cloud platform 1150; Example shared data center 1152; Example certifier 1210; Example delegation orchestration manager 1212; (multiple) example interfaces 1214; Example orchestration delegators 1216-1218; Example FMD 1310, 1420, 1422; Example cloud / edge 1320, 1410, 1412; Example FNeL 1340, 1430-1434; Example FN 1330-1336, 1440-1451; Example certification server 1501; and / or more generally, Figures 10-15The illustrated examples are edge infrastructures 1000, 1100, 1200 and / or fog infrastructures 1300, 1400. However, any other type of circuit system may be used additionally or alternatively, such as one or more analog or digital circuits, logic circuits, (multiple) programmable processors, (multiple) application-specific integrated circuits (ASICs), (multiple) programmable logic devices (PLDs), (multiple) field-programmable logic devices (FPLDs), (multiple) digital signal processors (DSPs), (multiple) coarse-grained reduced-precision architectures (CGRAs), (multiple) image signal processors (ISPs), etc. In some examples, the following are implemented by separate logic circuits: Example Master Orchestrator 1010, Example Auxiliary Orchestrators 1020-1024, Example Edge Entities 1030-1034, Example End-to-End Service Logic 1110, Example Application 1120, Example Edge Nodes 1122, 1222, Example NFV 1124, 1142, 1224, Example Edge SDK 1126, 1226, Example VIM 1128, 1148, 1228, Example MEC 1130, 1230; Example Shared Edge Infrastructure 1132, 1232; Example Edge Resources 1134-1136, 1154; Example LBO Switches 1138, 1238; Example Backhaul 1140; Example Control Plane 1144; Example Cloud Platform 1150; Example Shared Data Center 1152; Example Proofreader 1210; Example Delegated Orchestration Manager 1212; (Multiple) Example Interfaces 1214; Example Orchestration Delegators 1216-1218; Example FMD 1310, 1420, 1422; Example Cloud / Edge 1320, 1410, 1412; Example FNeL 1340, 1430-1434; Example FN Examples 1330-1336, 1440-1451, example verification server 1501, and / or more generally, example edge infrastructure 1000, 1100, 1200 and / or example fog infrastructure 1300, 1400. In some examples, example orchestrator manager 1212 implements the means for processing. In some examples, example orchestration delegates 1216-1218 implement the means for allocation. In some examples, example orchestration delegates 1216-1218 implement the means for monitoring.

[0179] although Figures 11-14 The diagram shows... Figure 10 The example implementation of the distributed computing infrastructure 1000, but Figures 11-14One or more of the elements, processes, and / or devices illustrated herein may be combined, split, rearranged, omitted, eliminated, and / or implemented in any other way. Further, the following may be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware: Example Master Orchestrator 1010, Example Auxiliary Orchestrators 1020-1024, Example Edge Entities 1030-1034, Example End-to-End Service Logic 1110, Example Application 1120, Example Edge Nodes 1122, 1222, Example NFV 1124, 1142, 1224, Example Edge SDK 1126, 1226, Example VIM 1128, 1148, 1228, Example MEC 1130, 1230; Example Shared Edge Infrastructure 1132, 1232; Example Edge Resources 1134-1136, 1154; Example LBO Switches 1138, 1238; Example Backhaul 1140; Example Control Plane 1144; Example Cloud Platform 1150; Example Shared Data Center 1152; Example Proofreader 1210; Example Delegated Orchestration Manager 1212; (Multiple) Example Interfaces 1214; Example Orchestration Delegators 1216-1218; Example FMD 1310, 1420, 1422; Example Cloud / Edge 1320, 1410, 1412; Example FNeL 1340, 1430-1434; Example FN 1330-1336, 1440-1451, example verification server 1501, and / or more generally, example edge infrastructure 1000, 1100, 1200 and / or example fog infrastructure 1300, 1400.Therefore, for example, any of the following can be implemented by one or more analog or digital circuits, logic circuits, (multiple) programmable processors, (multiple) programmable controllers, (multiple) graphics processing units (GPUs), (multiple) digital signal processors (DSPs), (multiple) application-specific integrated circuits (ASICs), (multiple) programmable logic devices (PLDs), and / or (multiple) field-programmable logic devices (FPLDs): Example Master Orchestrator 1010, Example Auxiliary Orchestrators 1020-1024, Example Edge Entities 1030-1034, Example End-to-End Service Logic 1110, Example Application 1120, Example Edge Nodes 1122, 1222, Example NFV 1124, 1142, 1224, Example Edge SDK 1126, 1226, Example VIM 1128, 1148, 1228, Example MEC 1130, 1230; Example Shared Edge Infrastructure 1132, 1232; Example Edge Resources 1134-1136, 1154; Example LBO Switches 1138, 1238; Example Backhaul 1140; Example Control Plane 1144; Example Cloud Platform 1150; Example Shared Data Center 1152; Example Proofreader 1210; Example Delegated Orchestration Manager 1212; (Multiple) Example Interfaces 1214; Example Orchestration Delegators 1216-1218; Example FMD 1310, 1420, 1422; Example Cloud / Edge 1320, 1410, 1412; Example FNeL 1340, 1430-1434; Example FN 1330-1336, 1440-1451, example verification server 1501, and / or more generally, example edge infrastructure 1000, 1100, 1200 and / or example fog infrastructure 1300, 1400.When reading any of the device or system claims of this patent covering pure software and / or firmware implementations, at least one of the following is thereby expressly defined as including a non-transient computer-readable storage device or disk (such as a memory, digital versatile disc (DVD), compact disc (CD), Blu-ray disc, etc.) containing such software and / or firmware: example master arranger 1010, example auxiliary arrangers 1020-1024, example edge entities 1030-1034, example end-to-end service logic 1110, example application 1120, example edge nodes 1122, 1222, example NFV 1124, 1142, 1224, example edge SDK 1126, 1226, example VIM 1128, 1148, 1228, example MEC 1130, 1230; Example Shared Edge Infrastructure 1132, 1232; Example Edge Resources 1134-1136, 1154; Example LBO Switches 1138, 1238; Example Backhaul 1140; Example Control Plane 1144; Example Cloud Platform 1150; Example Shared Data Center 1152; Example Proofreader 1210; Example Delegated Orchestration Manager 1212; (Multiple) Example Interfaces 1214; Example Orchestration Delegators 1216-1218; Example FMD 1310, 1420, 1422; Example Cloud / Edge 1320, 1410, 1412; Example FNeL 1340, 1430-1434; Example FN 1330-1336, 1440-1451, example verification server 1501, and / or more generally, example edge infrastructure 1000, 1100, 1200 and / or example fog infrastructure 1300, 1400. Furthermore, ... Figures 10-14 Example network infrastructures such as 1000, 1100, 1200, 1300, and 1400 can include, but are not limited to, [other types of infrastructure]. Figures 10-15 One or more elements, processes and / or devices other than those illustrated herein, or as Figures 10-15 The illustrations may include one or more alternative elements, processes, and / or devices, and / or may include any or all of more than one of the illustrated elements, processes, and devices. As used herein, the phrase “to communicate” (including its various variations) includes direct communication and / or indirect communication via one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or continuous communication, but additionally includes selective communication performed at periodic intervals, predetermined intervals, non-periodic intervals, and / or one-off events.

[0180] exist Figures 16-17 The diagram shows the representation used for implementation. Figures 10-14The example distributed network computing infrastructure 1000-1400 includes example hardware logic, machine-readable instructions, hardware-implemented state machines, and / or any combination thereof flowcharts. Machine-readable instructions can be one or more executable programs or portions thereof for execution by a computer processor and / or processor circuitry, such as those described below. Figure 18 The processor 1812 (and / or) shown in the example processor platform 1800 discussed above Figure 8A Example compute node 800 Figure 8B Examples include edge computing node 850, etc. The program may be embodied in software stored on a non-transitory computer-readable storage medium such as a CD-ROM, floppy disk, hard drive, DVD, Blu-ray disc, or memory associated with the processor 1812; however, the entire program and / or portions thereof may alternatively be executed by a device other than the processor 1812, and / or embodied in firmware or dedicated hardware. Furthermore, although references... Figures 16-17 The flowcharts illustrated in the diagrams describe example programs, but many other methods for implementing example distributed network computing infrastructure 1000-1400 can be used alternatively. For example, the execution order of the boxes can be changed, and / or some boxes in the described boxes can be changed, eliminated, or combined. Additionally or alternatively, any or all boxes in the diagrams can be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) configured to perform the corresponding operations without executing software or firmware. Processor circuitry can be distributed across different network locations and / or located locally on one or more devices (e.g., multi-core processors in a single machine, multiple processors distributed across a server rack, etc.).

[0181] The machine-readable instructions described herein can be stored in one or more of the following formats: compressed, encrypted, segmented, compiled, executable, and packaged. The machine-readable instructions described herein can be stored as data or data structures (e.g., portions of instructions, code, code representations, etc.) that can be used to create, manufacture, and / or produce machine-executable instructions. For example, machine-readable instructions can be segmented and stored on one or more storage devices and / or computing devices (e.g., servers) located in the same or different locations within a network or network set (e.g., in the cloud, at edge devices, etc.). Machine-readable instructions may involve one or more of the following: installation, modification, adaptation, updating, combination, supplementation, configuration, decryption, decompression, unpacking, distribution, reallocation, compilation, etc., such that they are directly readable, interpretable, and / or executable by computing devices and / or other machines. For example, machine-readable instructions can be stored in multiple parts that are individually compressed, encrypted, and stored on separate computing devices, wherein these parts, when decrypted, decompressed, and combined, form a set of executable instructions that implement one or more functions of a program as described herein.

[0182] In another example, machine-readable instructions may be stored in a state where they can be read by processor circuitry, but libraries (e.g., dynamic link libraries (DLLs)), software development kits (SDKs), application programming interfaces (APIs), etc., need to be added to execute the instructions on a specific computing device or other device. In yet another example, the machine-readable instructions (e.g., storage settings, data input, recorded network addresses, etc.) may need to be configured before they can be executed, either wholly or partially, and / or their corresponding programs. Therefore, as used herein, machine-readable media may include machine-readable instructions and / or programs, regardless of their specific format or state at storage or otherwise in a static or transit state.

[0183] The machine-readable instructions described in this article can be represented by any past, present, or future instruction language, scripting language, programming language, etc. For example, machine-readable instructions can be represented by any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, Hypertext Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0184] As mentioned above, executable instructions (e.g., computer and / or machine-readable instructions) stored on a non-transient computer and / or machine-readable medium can be used to implement [the system]. Figure 16 and / or Figure 17Example processes refer to non-transient computer- and / or machine-readable media such as: hard disk drives, flash memory, read-only memory, compact disks, digital multifunction disks, caches, random access memory, and / or any other storage device or disk that stores information therein for any duration (e.g., over an extended period, permanently, during a short instance, during temporary buffering and / or information caching). As used herein, the term non-transient computer-readable media is explicitly defined to include any type of computer-readable storage device and / or disk, excluding propagated signals and transmission media.

[0185] "Comprising" and "including" (and all their forms and tenses) are used herein as open-ended terms. Therefore, whenever a claim uses any form of "comprising" and "including" (e.g., including, comprising, including, containing, having, etc.) as a preamble or in the content of any kind of claim, it is to be understood that additional elements, items, etc., may be present without falling outside the scope of the corresponding claim or statement. As used herein, when the phrase "at least" is used as a transitional term, for example, in the preamble of a claim, it is open-ended in the same way that the terms "comprising" and "including" are open-ended. When used, for example, in forms such as A, B, and / or C, the term "and / or" refers to any combination or subset of A, B, C, such as (1) A alone, (2) B alone, (3) C alone, (4) A and B, (5) A and C, (6) B and C, and (7) A and B and C. As used herein in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A and B” is intended to refer to an implementation that includes any one of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A or B” is intended to refer to an implementation that includes any one of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. As used herein in the context of describing the implementation or execution of processes, instructions, actions, activities, and / or steps, the phrase “at least one of A and B” is intended to refer to an implementation that includes any one of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein in the context of describing the implementation or execution of processes, instructions, actions, activities and / or steps, the phrase “at least one of A or B” is intended to refer to an implementation that includes any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B.

[0186] As used herein, singular references (e.g., "a (a, an)", "first", "second", etc.) do not exclude plurals. The term "a (a or an)" as used herein refers to one or more of that entity. The terms "a (a)" (or "a (an)"), "one or more", and "at least one" are used interchangeably herein. Furthermore, although listed separately, multiple means, elements, or method actions may be implemented by, for example, a single unit or processor. Additionally, while individual features may be included in different examples or claims, these features may be combined, and inclusion in different examples or claims does not imply that the combination of features is infeasible and / or not advantageous.

[0187] Figure 16 This indicates that it can be executed to achieve [the desired result]. Figures 10-15 A flowchart of example machine-readable instructions for an example network computing infrastructure 1000. Figure 16 Example process 1600, as illustrated in the diagram, begins when example master orchestrator 1010 processes contracts and / or other requests for the execution of a service (Box 1610). For example, master orchestrator 1010 processes contracts that combine dynamic tracing of software development to provide an SLA for the execution of the software with one or more associated SLOs, and so on. For example, master orchestrator 1010 processes constraints, parameters, etc., associated with requests for resources involved in determining the execution of a service. In some examples, information such as contract requirements, constraints, service parameters, etc., may be organized as tokens.

[0188] The master orchestrator 1010 distributes the execution of contracts and their services among resources via auxiliary orchestrators 1020-1024 (Box 1620). For example, the master orchestrator 1010 communicates tasks associated with the execution of services to one or more auxiliary orchestrators 1020-1024 to orchestrate or manage the execution of services using hardware and / or software computing resources in the networked computing infrastructure 1000-1400. In some examples, the master orchestrator 1010 communicates tokens indicating the state of contracts and / or associated services, tasks(s) to be executed by a particular resource, other parameters, constraints, etc.

[0189] Multiple auxiliary orchestrators 1020-1024 monitor the execution of services and / or associated contracts (Box 1630). For example, multiple auxiliary orchestrators 1020-1024 monitor the execution of services and / or multiple service tasks on edge, cloud, and / or fog computing resources in their associated domains 1030-1034. Multiple auxiliary orchestrators 1020-1024 may compare task / service execution with one or more SLOs / SLAs associated with the same service contract to determine whether the execution meets the contractual constraints / requirements (Box 1640).

[0190] When execution fails to meet contractual constraints / requirements (e.g., appears to break SLOs, SLAs, etc. associated with the service contract), the (multiple) auxiliary orchestrators 1020-1024 may adjust the allocation of resources within the contract (Box 1650). For example, the first auxiliary orchestrator 1020-1024 may assign or dispatch the service and / or the (multiple) tasks associated with that service to the second auxiliary orchestrator 1020-1024. In some examples, tokens may be updated by the first auxiliary orchestrator 1020-1024 to reflect what has been completed by and passed from the first auxiliary orchestrator 1020-1024 to the second auxiliary orchestrator 1020-1024, so that the second auxiliary orchestrator 1020-1024 understands what needs to be done.

[0191] As shown in box 1630, control continues to monitor execution until the service has been executed and the contract is complete (box 1660). Upon completion, Figure 16 The example process 1600 illustrated in the diagram terminates at the end, but may be repeated in response to subsequent requests.

[0192] Figure 17 It provides information on the execution of contracts and their services allocated among resources via auxiliary orchestrators 1020-1024. Figure 16 A flowchart of further example details for example box 1620. Example Delegation Orchestration Manager 1212 ( Figure 12 The delegate orchestration manager 1212 receives requests to perform services (e.g., in conjunction with an SLA for software execution and one or more associated SLOs, in conjunction with dynamic tracing for software development, etc.) (Box 1710). For example, the delegate orchestration manager 1212 receives requests from the master orchestrator 1010 and / or other sources to perform applications or other services in accordance with constraints / parameters specified in a contract or SLA. For example, constraints, parameters, status, assignments, etc., may be represented in tokens provided by the master orchestrator 1010.

[0193] In response to a request, the example delegation orchestration manager 1212 generates orchestration delegations 1216-1218 (Box 1720). For example, delegation orchestration manager 1212 generates binary executables representing orchestration delegations 1216-1218. Example verifier 1210 verifies or otherwise confirms the authenticity, credibility, source, integrity, etc., of the files (Box 1730). For example, verifier 1210 may communicate with a verification server to calculate and verify hashes and / or other signatures associated with the files and their sources. After verification, the example delegation orchestration manager 1212 deploys example orchestration delegations 1216-1218 to manage resource allocation and service execution (Box 1740). For example, control is returned to Box 1630 to monitor service execution and resource allocation for entities 1030-1034 associated with orchestrators 1020-1024. Example process 1620 can be repeated when additional services, tasks, etc. are deployed for execution.

[0194] Figure 18 It is constructed for execution Figures 16-17 Instructions to achieve Figures 10-15 The example network computing infrastructure 1000 is illustrated in the block diagram of the example processor platform 1100. The processor platform 1800 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smartphone, a tablet device such as an iPad™), an internet device, a game console, a headset or other wearable device, or any other type of computing device.

[0195] The illustrated example processor platform 1800 includes a processor 1812. The illustrated example processor 1812 is hardware. For example, the processor 1812 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor can be a semiconductor-based (e.g., silicon-based) device. In this example, the processor 1812 implements an example auxiliary orchestrator 1022. The example auxiliary orchestrator 1022 includes an example validator 1210, an example delegation orchestration manager 1212, multiple example interfaces 1214, and example orchestration delegates 1216-1218. The example processor 1812 can similarly implement example auxiliary orchestrators 1020, 1024, an example master orchestrator 1010, etc.

[0196] The illustrated processor 1812 includes local memory 1813 (e.g., cache). The illustrated processor 1812 communicates via bus 1818 with main memory, which includes volatile memory 1814 and non-volatile memory 1816. Volatile memory 1814 may be implemented using synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), RAMBUS® dynamic random access memory (RDRAM®), and / or any other type of random access memory device. Non-volatile memory 1816 may be implemented using flash memory and / or any other desired type of memory device. Access to main memory 1814 and 1816 is controlled by a memory controller.

[0197] The processor platform 1800 illustrated also includes interface circuitry 1820. Interface circuitry 1820 can be implemented using any type of interface standard, such as an Ethernet interface, Universal Serial Bus (USB), Bluetooth® interface, Near Field Communication (NFC) interface, and / or PCI Express (PCI Fast) interface.

[0198] In the illustrated example, one or more input devices 1822 are connected to interface circuitry 1820. The input devices 1822 allow the user to input data and / or commands into processor 1812. The input devices may be implemented as, for example, audio sensors, microphones, (still or video) cameras, keyboards, buttons, mice, touchscreens, trackpads, trackballs, isotope mice, and / or voice recognition systems.

[0199] One or more output devices 1824 are also connected to the interface circuitry 1820 of the illustrated example. The output devices 1824 may be implemented, for example, by display devices (e.g., light-emitting diodes (LEDs), organic light-emitting diodes (OLEDs), liquid crystal displays (LCDs), cathode ray tube displays (CRTs), in-plane switching (IPS) displays, touchscreens, etc.), haptic output devices, printers, and / or speakers. Therefore, the interface circuitry 1820 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0200] The illustrated interface circuit 1820 also includes communication devices such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces to facilitate the exchange of data with external machines (e.g., any kind of computing device) via network 1826. Communication can be via, for example, Ethernet connections, digital subscriber line (DSL) connections, telephone line connections, coaxial cable systems, satellite systems, line-to-line wireless systems, cellular telephone systems, etc.

[0201] The illustrated processor platform 1800 also includes one or more mass storage devices 1828 for storing software and / or data. Examples of such mass storage devices 1828 include floppy disk drives, hard disk drives, compact disk drives, Blu-ray disc drives, redundant array of independent disks (RAID) systems, and digital multifunction disc (DVD) drives.

[0202] Figures 16-17 The machine-executable instructions 1832 can be stored in mass storage device 1828, in volatile memory 1814, in non-volatile memory 1816, and / or on a removable non-transient computer-readable storage medium such as a CD or DVD.

[0203] As will be understood from the foregoing, example methods, apparatuses, systems, and artifacts for implementing dynamic management of services and / or other processes on multi-edge network computing infrastructures have been disclosed. Some examples establish orchestrator meshes for sharing in the execution of services via edge computing platforms. Some examples provide orchestrators within the orchestrator mesh to coordinate the allocation, monitoring, and tuning of applications, processes, and other services across network infrastructure and across devices, constructs, and privilege boundaries. The disclosed methods, apparatuses, systems, and artifacts improve the efficiency of using computing devices by enabling efficient and effective monitoring and management of resources across distributed systems at the cloud, edge, and fog levels. The disclosed methods, apparatuses, and artifacts accordingly relate to one or more improvements in computing capabilities.

[0204] Some examples implement an orchestrator mesh to monitor the application's SLA during application execution and compare the monitored SLA data with the provided SLA. Some examples enable orchestrators to distribute and / or otherwise reallocate resources among orchestrators to help ensure that the provided SLA is met by the executing application. For example, memory allocation can be dynamically adjusted across edge boundaries to accommodate application and / or other service execution. Some examples instantiate orchestrators or delegated orchestration modules in memory, as part of the compute platform hardware, on local edge nodes or devices, etc. In some examples, orchestrators execute on FPGAs and / or accelerators running on compute entities to manage services dispatched to edge devices on behalf of those compute entities. For example, tokens can be used to enable one orchestrator to pass instructions to another orchestrator indicating what has been done and what remains to be done. Some examples enable negotiation (e.g., facilitated by a master edge orchestrator, etc.) between orchestrators to determine which orchestrator is responsible for disposing of which task for application / service execution.

[0205] Although certain example methods, apparatuses, and articles have been disclosed herein, the scope of this patent is not limited thereto. Rather, this patent covers all methods, apparatuses, and articles that fall within the scope of the examples herein.

[0206] This paper discloses example methods, apparatuses, systems, and artifacts for dynamically controlling the orchestration of grids in cloud-edge computing environments. Further examples and combinations thereof include the following:

[0207] Example 1 is an orchestrator apparatus comprising: an interface for receiving a service to be executed and monitoring the execution of the service; an orchestration delegation for selecting resources and orchestrating the deployment of the service using the selected resources; and a delegation orchestration manager for managing the orchestration delegation based on: (a) information from the interface, and (b) information corresponding to the execution of the service by the orchestration delegation.

[0208] Example 2 includes Example 1, wherein the orchestrator device communicates with an edge orchestrator to receive instructions from the edge orchestrator relating to the execution of a service.

[0209] Example 3 includes Example 2, wherein the orchestrator device is a portion of the orchestrator mesh managed by the edge orchestrator.

[0210] Example 4 includes Example 1, where orchestration delegation and delegation orchestration manager are used to adjust resource allocation to conform to the contract associated with the execution of the service.

[0211] Example 5 includes Example 4, where adjustments to resource allocation are made for cross-edge computing entities to occur.

[0212] Example 6 includes Example 1, wherein the resource is implemented in at least one of edge, fog, or cloud computing entities.

[0213] Example 7 includes Example 1, wherein the orchestrator device is implemented in at least one of an edge, fog, or cloud computing entity.

[0214] Example 8 includes Example 1, wherein orchestration delegation is used to be instantiated on a first computing entity, and wherein a service is deployed to perform using resources on a second computing entity.

[0215] Example 9 includes Example 1, where orchestration delegation is used to instantiate using a verified binary file.

[0216] Example 10 includes Example 1, and further includes multiple orchestration delegations for orchestrating multiple services.

[0217] Example 11 includes Example 1, and further includes multiple interfaces for receiving a service to be executed, for monitoring the execution of the service, and for retrieving historical information to evaluate performance over time.

[0218] Example 12 includes Example 1 and further includes a validator for verifying orchestration delegation.

[0219] Example 13 is at least one non-transient computer-readable storage medium including instructions that, when executed, cause at least one processor to at least: process a request to perform a service in an edge cloud infrastructure; allocate the performance of the service among resources via at least one orchestrator; monitor the performance of the service; and, when the monitored performance of the service fails to satisfy the request, use at least one orchestrator to adjust the allocation of resources for performing the service.

[0220] Example 14 includes Example 13, wherein at least one processor is a portion of an orchestrator mesh managed by an edge orchestrator, and wherein instructions, when executed, cause at least one processor to communicate with the edge orchestrator to adjust the allocation of resources for performing services.

[0221] Example 15 includes Example 13, wherein at least one processor allocates the execution of a service among resources by at least the following: generating an orchestration delegation to select resources and orchestrate the deployment of the service using the selected resources; and deploying an orchestration delegation to manage the allocation of resources and the execution of the service.

[0222] Example 16 includes Example 15, wherein at least one processor is used to distribute the execution of a service among resources at least by verifying an orchestration delegation, the orchestration delegation being implemented in at least one of an edge, fog, or cloud computing entity.

[0223] Example 17 is a method for orchestrating the execution of a service in an edge computing environment, the method comprising: processing a request to execute a service in an edge cloud infrastructure by executing instructions using at least one processor; allocating the execution of the service among resources via at least one orchestrator by executing instructions using at least one processor; monitoring the execution of the service; and adjusting the allocation of resources for executing the service using at least one orchestrator when the monitored execution of the service does not satisfy the request.

[0224] Example 18 includes Example 17, wherein at least one processor is a portion of an orchestrator mesh managed by an edge orchestrator, and wherein adjustments to the allocation of resources occur while communicating with the edge orchestrator.

[0225] Example 19 includes Example 17, wherein allocating the execution of a service among resources includes: generating an orchestration delegation to select resources and using the selected resources to orchestrate the deployment of the service; and deploying the orchestration delegation to manage the allocation of resources and the execution of the service.

[0226] Example 20 includes Example 19, wherein allocating the execution of a service among resources includes verifying an orchestration delegation that is implemented in at least one of an edge, fog, or cloud computing entity.

[0227] Example 21 is an edge resource orchestrator that includes a memory circuitry for storing instructions and includes at least one processor for executing instructions to at least: process requests for performing services in an edge cloud infrastructure; allocate the execution of the service among resources; monitor the execution of the service; and adjust the allocation of resources for performing the service when the monitored execution of the service does not satisfy the request.

[0228] Example 22 includes Example 21, wherein the edge resource orchestrator is an auxiliary orchestrator for a portion of the orchestrator mesh managed by the edge orchestrator, and wherein, when executed, instructions cause at least one processor to communicate with the main orchestrator to adjust the allocation of resources for performing services.

[0229] Example 23 includes Example 22, wherein at least one processor allocates the execution of a service among resources by at least the following: generating an orchestration delegation to select resources and orchestrate the deployment of the service using the selected resources; and deploying an orchestration delegation to manage the allocation of resources and the execution of the service.

[0230] Example 24 includes Example 23, wherein at least one processor is used to distribute the execution of a service among resources at least by verifying an orchestration delegation, the orchestration delegation being implemented in at least one of an edge, fog, or cloud computing entity.

[0231] Example 25 is a master orchestrator that communicates with a grid of auxiliary orchestrators to allocate resources for executing services in an edge cloud environment. The master orchestrator includes a memory circuitry for including instructions and includes at least one processor for executing the instructions to at least: process requests for executing services in an edge cloud infrastructure; allocate the execution of the service among the auxiliary orchestrator grids; monitor the execution of the service; and adjust the allocation of the execution of the service among the auxiliary orchestrator grids when the monitored execution of the service does not satisfy the request.

[0232] Example 26 is an edge device that includes a first edge node for performing a service and includes an orchestrator for: processing requests for performing the service; allocating the performance of the service to edge nodes; monitoring the performance of the service; and adjusting the allocation of the performance of the service to a second edge node when the monitored performance of the service does not satisfy the request.

[0233] Example 27 includes Example 26, wherein the edge device is a first edge device, and wherein the second edge node is located on the second edge device.

[0234] Example 28 is an apparatus comprising: means for processing requests for performing services in an edge cloud infrastructure; means for allocating the performance of services among resources; and means for monitoring the performance of services, wherein the means for allocating is used to adjust the allocation of resources for performing services when the monitored performance of a service does not satisfy a request.

[0235] In Example 29, the subject matter as described in any one or more of Examples 1-28 includes a combination of edge, cloud, and fog resources.

[0236] In Example 30, the subject matter described in any one or more of Examples 1-28 includes a combination of one or more edge orchestrators, cloud orchestrators, and fog orchestrators.

[0237] Example 31 includes any one or more of Examples 1-12, and further includes multiple orchestration delegations arranged across one or more devices, constructions, or privilege boundaries.

[0238] Example 32 is a memory infrastructure for implementing any of the dynamically allocated ones in Examples 1-31.

[0239] Example 33 includes any one of Examples 1-31, wherein during execution, a service level agreement is monitored and compared with a provided service level agreement to achieve at least one of resource distribution or reallocation, thereby aligning the monitored service level agreement with the provided service level agreement.

[0240] Example 34 is an edge computing node that includes a processing circuitry system for performing any of the items in Examples 17-20.

[0241] Example 35 is a system comprising: means for processing requests for performing services in an edge cloud infrastructure; means for allocating the performance of the services among resources via at least one orchestrator; means for monitoring the performance of the services; and means for adjusting the allocation of resources for performing the services when the monitored performance of the services does not satisfy the request.

[0242] Example 36 includes Example 35, wherein the means form a portion of an orchestrator mesh managed by an edge orchestrator, and wherein means for adjusting the allocation of resources communicate with the edge orchestrator.

[0243] Example 37 includes any one of Examples 35-36, wherein the means for allocating the execution of a service among resources includes allocating the resources by: generating an orchestration delegation to select resources and using the selected resources to orchestrate the deployment of the service; and deploying the orchestration delegation to manage the allocation of resources and the execution of the service.

[0244] Example 38 includes any one of Examples 35-37, wherein the means for allocating the execution of a service among resources performs the allocation by verifying an orchestration delegation implemented in at least one of an edge, fog, or cloud computing entity.

[0245] The appended claims are hereby incorporated into the detailed description by reference, wherein each claim is itself a separate embodiment of the present disclosure.

Claims

1. A computing system configured to provide at least one microservice to at least one client device via at least one network, the at least one client device being associated with at least one of a plurality of subscribers of the computing system, the at least one client device being configured to provide request data to the computing system via the at least one network, the request data being used to request the provision of the at least one microservice to the at least one client device, the computing system comprising: Distributed processing resources, including artificial intelligence (AI) processing acceleration resources, including graphics processing unit (GPU) accelerators; as well as A resource management system is configured to perform operations, including: At least one allocation of at least one portion of the distributed processing resource to execute at least one functional instance associated with the at least one microservice is determined based on the following: (1) the request data, (2) the accelerated resource availability, (3) the functional extension data, and (4) the subscriber agreement data. Monitor the execution of the at least one functional instance to determine whether the execution of the at least one functional instance meets the required data; and Based on the monitoring, determine whether to adjust at least one allocation of at least one portion of the distributed processing resources to allow the execution of at least one functional instance to comply with the required data; in: The subscriber agreement data is associated with at least one of the plurality of subscribers of the computing system; The computing system is used to provide result data based on the execution results of the at least one functional instance, the result data indicating at least partially the execution results of the at least one functional instance to the at least one client device via the at least one network; The at least one functional instance is used to implement at least partially using at least one container environment; The AI ​​processing acceleration resources can be configured to at least partially implement at least one AI processing task; and The computing system can be configured to provide at least in part the security and isolation of the containerized environment based on the subscriber agreement data.

2. The computing system as described in claim 1, characterized in that: The computing system can be configured to expand functionality on demand, based on workload requirements, and / or as needed.

3. The computing system as described in claim 2, characterized in that: The distributed processing resources are at least partially included in the one or more servers; The one or more servers are included in one or more data centers; and The request data can be configured to be associated with workload description information.

4. The computing system as described in claim 3, characterized in that: The at least one AI processing task may be configured to be associated with one or more of the following: One or more inference operations; One or more machine learning operations; One or more training operations; One or more classification operations; One or more visual data processing operations; One or more neural network processing operations; Transportation data processing; Internet of Things (IoT) data processing; and / or One or more object detection operations.

5. The computing system as described in claim 4, characterized in that: The resource management system is used to be managed at least in part via at least one application programming interface (API); and The computing system includes a cloud computing system.

6. A method implemented using a computing system, the computing system being configured to provide at least one microservice to at least one client device via at least one network, the at least one client device being associated with at least one subscriber of a plurality of subscribers of the computing system, the at least one client device being configured to provide request data to the computing system via the at least one network, the request data being used to request the provision of the at least one microservice to the at least one client device, the computing system including distributed processing resources and a resource management system, the distributed processing resources including artificial intelligence (AI) processing acceleration resources, the AI ​​processing acceleration resources including graphics processing units (GPUs), the method comprising: The operation is performed by the resource management system, and the operation includes: At least one allocation of at least one portion of the distributed processing resource to execute at least one functional instance associated with the at least one microservice is determined based on the following: (1) the request data, (2) the accelerated resource availability, (3) the functional extension data, and (4) the subscriber agreement data. Monitor the execution of the at least one functional instance to determine whether the execution of the at least one functional instance meets the required data; and Based on the monitoring, determine whether to adjust at least one allocation of at least one portion of the distributed processing resources to allow the execution of at least one functional instance to comply with the required data; in: The subscriber agreement data is associated with at least one of the plurality of subscribers of the computing system; The computing system is used to provide result data based on the execution results of the at least one functional instance, the result data indicating at least partially the execution results of the at least one functional instance to the at least one client device via the at least one network; The at least one functional instance is used to implement at least partially using at least one container environment; The AI ​​processing acceleration resources can be configured to at least partially implement at least one AI processing task; and The computing system can be configured to provide at least in part the security and isolation of the containerized environment based on the subscriber agreement data.

7. The method as described in claim 6, characterized in that: The computing system can be configured to expand functionality on demand, based on workload requirements, and / or as needed.

8. The method as described in claim 7, characterized in that: The distributed processing resources are at least partially included in the one or more servers; The one or more servers are included in one or more data centers; and The request data can be configured to be associated with workload description information.

9. The method as described in claim 8, characterized in that: The at least one AI processing task may be configured to be associated with one or more of the following: One or more inference operations; One or more machine learning operations; One or more training operations; One or more classification operations; One or more visual data processing operations; One or more neural network processing operations; Transportation data processing; Internet of Things (IoT) data processing; and / or One or more object detection operations.

10. The method as described in claim 9, characterized in that: The resource management system is used to be managed at least in part via at least one application programming interface (API); and The computing system includes a cloud computing system.

11. A machine-readable storage medium storing at least one instruction, which, when executed by at least one machine, causes the execution of the method of any one of claims 6 to 10.

Images