Multi-entity Resource, Security, and Service Management in Edge Computing Deployments

Abstract

Various aspects of methods, systems, and use cases are disclosed for multi-entity (e.g., multi-tenant) edge computing deployments. Among other things, various configurations and features enable resource management (e.g., hardware control and coordination, acceleration, network, resource usage handling), security (e.g., secure execution and communication, isolation, contention), and service management (e.g., coordination, connectivity, workload coordination) in edge computing deployments, such as with multiple edge nodes in an edge computing environment configured to execute workloads among multiple tenants.

Description

[Background technology]

[0001] Priority This application claims priority to U.S. Provisional Patent Application No. 62 / 841,042, filed April 30, 2019; U.S. Provisional Patent Application No. 62 / 907,597, filed September 28, 2019; and U.S. Provisional Patent Application No. 62 / 939,303, filed November 22, 2019, all of which are incorporated by reference in their entireties.

[0002] background Edge computing, at a general level, refers to the implementation, coordination, and utilization of processing and resources near the "edge" or collection of "edges" of a network. The goals of this deployment are to improve total cost of ownership (especially compared to traditional cloud computing), reduce application and network latency, reduce network backhaul traffic and associated energy consumption, increase service capacity, and improve compliance with security or data privacy requirements. Components capable of performing edge computing operations ("edge nodes") can reside anywhere required by the system architecture or ad hoc services (e.g., in high-performance computing data centers or cloud facilities; designated edge node servers, enterprise servers, roadside servers, telecom central offices; or in a local or peer at-the-edge device providing a consuming edge service).

[0003] Applications applicable to edge computing include, but are not limited to, the virtualization of traditional network functions (e.g., to handle communications or internet services) and the introduction of next-generation functions and services (e.g., to support fifth-generation network services). Use cases expected to widely leverage edge computing include connected and autonomous vehicles, surveillance, data analysis for Internet of Things (IoT) devices, video encoding and analysis, location-aware services, device sensing in smart cities, and many other network- and compute-intensive services.

[0004] In some scenarios, edge computing can provide or host cloud-like distributed services, orchestrating and managing application and collaborative service instances across many types of storage and compute resources. Edge computing is also expected to be tightly integrated with existing use cases and technologies developed for IoT and fog / distributed network configurations, as endpoint devices, clients, and gateways seek to access network resources and applications closer to the edge of the network. [Brief explanation of the drawings]

[0005] In the drawings, which are not necessarily drawn to scale, like numbers may refer to like components in different drawings. Like numbers with different letter suffixes may represent different instances of like elements. Some embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0006] [Figure 1] This section provides an overview of the edge cloud configuration for edge computing.

[0007] [Figure 2]It represents the operational layer between endpoints, edge cloud, and cloud computing environments.

[0008] [Figure 3] This shows an overview of the distributed processing layer deployed between edge computing systems.

[0009] [Figure 4] This shows an overview of the distributed processing layer deployed between edge computing systems.

[0010] [Figure 5] Presents over-the-top and network aggregation approaches for networking and services in edge computing systems. [Figure 6] Presents over-the-top and network aggregation approaches for networking and services in edge computing systems.

[0011] [Figure 7] We demonstrate the deployment and orchestration of a virtual edge configuration across multiple edge nodes and an edge computing system operating among multiple tenants. [Figure 8] We demonstrate the deployment and orchestration of a virtual edge configuration across multiple edge nodes and an edge computing system operating among multiple tenants.

[0012] [Figure 9] This figure shows various processing configurations for deploying containers in edge computing systems.

[0013] [Figure 10A] This paper demonstrates the deployment of distributed edge computing using coordinated processing functions and services in edge computing systems. [Figure 10B]This paper demonstrates the deployment of distributed edge computing using coordinated processing functions and services in edge computing systems.

[0014] [Figure 11] This shows a configuration for edge computing that cooperates among edge meshes to provide an edge computing system.

[0015] [Figure 12] Demonstrates vehicular computing and communications use cases, including mobile access to applications in edge computing systems.

[0016] [Figure 13] A comparison of operational considerations for edge computing across operational network layers is presented.

[0017] [Figure 14] Shows placement and latency for edge computing between operational network layers.

[0018] [Figure 15] Shows the deployment and mapping of workloads to the operational layer of an edge computing system.

[0019] [Figure 16] Shows the mapping of workload types to service capabilities of edge computing systems.

[0020] [Figure 17] This figure shows the mapping of workload types to execution platforms in an edge computing system.

[0021] [Figure 18]This shows service operation for multiple tenants across multiple layers of edge computing hardware configuration in an edge computing system.

[0022] [Figure 19] Further mapping of edge computing hardware configurations to operational deployment and latency at the network layer is presented.

[0023] [Figure 20] Further mapping of use cases and workloads to operational deployments of edge computing hardware configurations is presented.

[0024] [Figure 21A] Further examples of edge computing hardware configurations based on operational objectives are presented. [Figure 21B] Further examples of edge computing hardware configurations based on operational objectives are presented. [Figure 21C] Further examples of edge computing hardware configurations based on operational objectives are presented.

[0025] [Figure 22A] Provides an overview of exemplary components for processing deployed on computing nodes within an edge computing system.

[0026] [Figure 22B] provides a further overview of exemplary components within a computing device in an edge computing system.

[0027] [Figure 22C] Provides a further overview of exemplary components within a mobile computing device within an edge computing system.

[0028] [Figure 22D]provides a further overview of exemplary components within a configurable server rack within an edge computing system.

[0029] [Figure 23] We demonstrate low connectivity clustering of edge nodes and connection coordination based on notifications from the Edge Heartbeat component.

[0030] [Figure 24] We demonstrate high-connectivity clusters of edge nodes and connection coordination based on notifications from the Edge Heartbeat component.

[0031] [Figure 25] An example edge deployment using an edge heartbeat component in the edge host is shown.

[0032] [Figure 26] An example of data flow between edge nodes is shown.

[0033] [Figure 27] We present techniques for data attestation in edge computing networks.

[0034] [Figure 28] A system diagram showing the architecture for edge service distribution is shown.

[0035] [Figure 29] We present a scenario for parallel data acceleration for edge computing systems.

[0036] [Figure 30] We present an example architecture that includes the building blocks for improved edge-to-cloud inference processing.

[0037] [Figure 31]1 shows a system diagram for implementing an Edge Services Page Table.

[0038] [Figure 32] Shows the Edge Services Page Table.

[0039] [Figure 33] A system diagram showing domain separation of orchestrators in different edge cloud locations is shown.

[0040] [Figure 34] A system diagram showing the edge cloud orchestrator managing the edge cloud region is shown.

[0041] [Figure 35] 1 shows a flow chart illustrating an exemplary broadcast and borrowing of resources between three different orchestrators.

[0042] [Figure 36] A system diagram of an architecture for aggregating data at edge devices is shown.

[0043] [Figure 37] An example of an edge gateway architecture is shown.

[0044] [Figure 38] We present an example of training and using a machine learning system for canned handoff.

[0045] [Figure 39] Demonstrates automatic service replication between edge computing nodes.

[0046] [Figure 40] 1 illustrates a block diagram of an example broadband-based and wireless-based system for customer premises equipment (CPE)-based performance forwarding.

[0047] [Figure 41] 1 illustrates a block diagram of an example system for CPE-based execution forwarding.

[0048] [Figure 42] 1 illustrates a flow diagram of an example technique for a policy-driven approach to achieving a predictable means of allocating computational power for CPE-based execution forwarding.

[0049] [Figure 43] Provides an example of an edge ecosystem domain architecture.

[0050] [Figure 44] An example of a virtualized multi-domain model is shown.

[0051] [Figure 45] We present a scenario for secure data sharing between edge computing nodes.

[0052] [Figure 46] FIG. 1 is a block diagram of an example system for group secure communication at the edge.

[0053] [Figure 47] A system diagram showing an overview of the edge system including a Competitive Analysis as a Service (CAaS) provider is shown.

[0054] [Figure 48] 1 shows a system diagram illustrating the components of a CAaaS provider.

[0055] [Figure 49] 1 illustrates an example architecture utilizing Device Identifier Composition Engine (DICE) layering and Composite Device Identifier (CDI) processing.

[0056] [Figure 50] An example architecture with DICE layering using asymmetric key generation is shown.

[0057] [Figure 51] 1 shows an example flow for implicit attestation of SLA parameters using DICE layering to generate workload-specific attestation keys.

[0058] [Figure 52] We present a system that provides an Edge Evidence Name Service (EANS) architecture.

[0059] [Figure 53] A system diagram of the edge architecture showing an accelerator accepting a curry function (e.g., bitstream) and a workload is shown.

[0060] [Figure 54] We present a system for multi-tenant edge cloud power management based on dynamic SLAs and billing criteria.

[0061] [Figure 55] 1 illustrates an example of a biometric hierarchical architecture.

[0062] [Figure 56] We present a scenario for SLA migration, monitoring, and propagation among multiple edge nodes in an edge computing system.

[0063] [Figure 57] Shows a system diagram for edge function-based changes to service level objectives.

[0064] [Figure 58] Shows a three-level hierarchy of an edge computing system between the cloud, edge, and mobile device locations.

[0065] [Figure 59] An example of edge service migration is shown.

[0066] [Figure 60] An example of partial edge service migration is shown.

[0067] [Figure 61] 1 illustrates an example of horizontal proxying for edge workloads.

[0068] [Figure 62] An example of moving an edge workload to multiple forward edge workloads based on workload component type is shown.

[0069] [Figure 63] Shows a system diagram for federated distributed edge telemetry services.

[0070] [Figure 64] This paper presents an implementation of a distributed system-to-system configuration that is configured to achieve load balancing within an edge computing system.

[0071] [Figure 65] An example of connectivity between different edge computing nodes is shown.

[0072] [Figure 66] An example of edge communication interaction between a Telecommunications Service Provider (TSP) and a Cloud Service Provider (CSP) is shown.

[0073] [Figure 67] 1 illustrates a 5G-MEC (Multi-Access Edge Computing) interconnection that can be deployed in an exemplary edge computing system.

[0074] [Figure 68] FIG. 1 is a simplified diagram of an overall 5G Next Generation (NG) system architecture that can be deployed in an exemplary edge computing system.

[0075] [Figure 69] 1 illustrates a non-roaming 5G system architecture that can be deployed in an exemplary edge computing system.

[0076] [Figure 70] Illustrates a 5G service-based architecture and a general MEC architecture that can be deployed in an exemplary edge computing system.

[0077] [Figure 71] An example illustrates the deployment of integrated MEC in a 5G network that can be used in an edge computing system.

[0078] [Figure 72] An exemplary mobile edge system reference architecture is shown.

[0079] [Figure 73] Examples of MEC deployment options at different edge computing system levels are presented.

[0080] [Figure 74] An exemplary illustration of MEC support in a 5G system that can be used in an edge computing system.

[0081] [Figure 75] 1 illustrates MEC mapping in a 5G system that can be used in an exemplary edge computing system. [Figure 76] 1 illustrates MEC mapping in a 5G system that can be used in an exemplary edge computing system. [Figure 77] 1 illustrates MEC mapping in a 5G system that can be used in an exemplary edge computing system.

[0082] [Figure 78] 1 illustrates the deployment of MEC in an exemplary 5G system.

[0083] [Figure 79] Figure 1 illustrates components of an exemplary 5G-NR architecture with Control Unit-Control Plane (CU-CP)-Control Unit-User Plane (CU-UP) separation related to network slicing.

[0084] [Figure 80] This diagram shows an example of a network slice that can be used in an edge computing deployment.

[0085] [Figure 81] An exemplary MEC network architecture supporting slice management, resource management, and traceability functions that can be deployed from an edge computing system is presented.

[0086] [Figure 82] This figure shows an MEC reference architecture in a network functions virtualization (NFV) environment that can be deployed from an exemplary edge computing system.

[0087] [Figure 83] Demonstrates management of the MEC platform as a virtualized network function that can be deployed from an exemplary edge computing system. DETAILED DESCRIPTION OF THE INVENTION

[0088] The following embodiments relate generally to implementing data processing, service management, resource allocation, computation management, network communications, application partitioning, and communications systems, and more particularly to techniques and configurations for adapting various edge computing devices and entities to dynamically support multiple entities (e.g., multiple tenants, users, stakeholders, service instances, applications, etc.) in a distributed edge computing environment.

[0089] In the following description, methods, configurations, and related apparatus are disclosed for various improvements to the configuration and functional capabilities of edge computing architectures and implemented edge computing systems. These improvements may benefit a variety of use cases, particularly those involving multiple stakeholders in the edge computing system, whether in the form of multiple users of the system, multiple tenants in the system, multiple devices or user equipment interacting with the system, multiple services provided by the system, multiple resources available or managed within the system, multiple forms of network access exposed for the system, multiple locations of operation for the system, etc. Such multi-dimensional aspects and considerations are generally referred to herein as "multi-entity" constraints, with specific discussion of resources managed or coordinated in multi-tenant and multi-service edge computing configurations.

[0090] In the exemplary edge network system described below, computing and storage resources are moved closer to the edge of the network (e.g., closer to clients, endpoint devices, or "things"). By moving computing and storage resources closer to the devices that generate or use data, various latency, compliance, and / or monetary or resource cost constraints can be achieved compared to standard network (e.g., cloud computing) systems. To do so, in some examples, pools of computing, memory, and / or storage resources may be located or otherwise equipped on local servers, routers, and / or other network equipment. Such local resources facilitate meeting constraints imposed on the system. For example, local computing and storage resources enable edge systems to perform computations in real time or near real time, which may be considered in low-latency use cases such as autonomous driving, video surveillance, and mobile media consumption. Additionally, these resources can benefit from service management at the edge systems, which provide the ability to scale and meet local SLAs, manage tiered service requirements, and enable local features and functionality on a temporary or permanent basis.

[0091] An exemplary edge computing system can support and / or provide various services to endpoint devices (e.g., client user equipment (UE)), each of which may have different requirements or constraints. For example, some services may have priority or quality of service (QoS) constraints (e.g., traffic data for autonomous vehicles may have a higher priority than temperature sensor data), reliability and resilience (e.g., traffic data may require mission-critical reliability, while temperature data may be tolerant of some error variance), as well as power, cooling, and form factor constraints. These and other technical constraints can introduce significant complexity and technical challenges when applied in a multi-stakeholder context.

[0092] This paper begins with a brief overview of terminology applicable to edge computing (Section I). This is followed by an overview of edge computing technologies and configurations, as well as an example use case for a multi-stakeholder service (Section II). The types of technical improvements provided by the use case (Section III.A) for the multi-stakeholder service (Section III) are then discussed, including improvements to (a) resource allocation and release (Section III.B), (b) security and privacy (Section III.C), and (c) orchestration and service levels (Section III.B), in addition to a discussion of the relevant technology domains. This is followed by a discussion of potential connectivity configurations for edge computing systems (Section IV), as well as a concluding discussion of example configurations and methods embodying the improvements described herein.

[0093] I. Terminology As used herein, the term "edge computing" encompasses many implementations of distributed computing that move processing activities and resources (e.g., computational, storage, and acceleration resources), including those typically performed as cloud processing activities or by cloud processing resources, to the "edge" of a network, thereby reducing latency and increasing throughput for end-point users (e.g., client devices, user equipment, etc.). Such edge computing implementations generally encompass providing such activities and resources in cloud-like services, functions, applications, and subsystems from one or more locations accessible via a wireless network. Accordingly, references to the "edge" of a network, cluster, domain, system, or computer configuration as used herein refer to a group or grouping of functionally distributed computing elements and, therefore, generally do not relate to an "edge" (link or connection) as used in graph theory.

[0094] The specific configuration of edge computing applications and services accessible over mobile wireless networks (e.g., cellular and Wi-Fi data networks) is referred to as "mobile edge computing" or "multi-access edge computing," which may be referred to by the acronym "MEC." The use of "MEC" in this application may refer to the standardized implementation (ETSI MEC) promulgated by the European Telecommunications Standards Institute (ETSI). Terms used by the ETSI MEC specification are generally incorporated herein by reference, unless a conflicting definition or usage is given herein.

[0095] As used herein, the terms "computing node" or "computing device" refer to an identifiable entity that implements aspects of edge computing operations, whether it is part of a larger system, a collection of distributed systems, or a standalone device. In some examples, a computing node may be referred to as an "edge node," "edge device," or "edge system," whether it operates as a client, server, or intermediate entity. Particular implementations of a computing node may be incorporated into a server, base station, gateway, roadside unit, on-premise, UE, or end-consumer device, etc. Furthermore, a computing node or computing device may include different types or classes of hardware, or configurations of such hardware, based on the resources available to the node or device (e.g., power, processing, space, temperature, and other operational considerations or constraints). Thus, many variations of hardware are intended to be encompassed by a computing node or computing device.

[0096] As used herein, the term "base station" refers to a network element in a radio access network (RAN), such as a fourth-generation (4G) or fifth-generation (5G) mobile communications network, that transmits and receives radio signals within one or more cells to and from user equipment (UE). A base station may have an integrated antenna or may be connected to an antenna array by a feeder cable. A base station uses specialized digital signal processing and network function hardware. In some examples, a base station may be divided into multiple software-driven functional blocks for flexibility, monetary or resource cost, and performance. In some examples, a base station may include an evolved node B (eNB) or next-generation node B (gNB). In some examples, a base station may operate or include computer hardware to operate as a computing node. However, in many scenarios discussed herein, a RAN base station may be replaced by an access point (e.g., a radio network access point) or other network access hardware.

[0097] As used herein, the term "central office" (or CO) refers to a point of aggregation of telecommunications infrastructure within an accessible or defined geographic area, where telecommunications service providers have often traditionally located switching equipment for one or more types of access networks. A CO can be physically designed to house telecommunications infrastructure facilities, computers, data storage, and network resources. However, a CO need not be a location designated by a telecommunications service provider. A CO can host any number of computing devices for edge applications and services, or even for local implementations of cloud-like services.

[0098] As used herein, the term "cloud service provider" (or CSP) refers to an organization that operates "cloud" resources, typically at a large scale, consisting of centralized, regional, and edge data centers (e.g., as used in the context of public clouds). In other instances, a CSP may also be referred to as a cloud service operator (CSO). Generally, references to "cloud computing" refer to computing resources and services provided by a CSP or CSO in remote locations with at least some of the increased latency, distance, or constraints associated with edge computing.

[0099] As used herein, the term "data center" refers to a purpose-built structure intended to house multiple high-performance computing and data storage nodes, such that a large amount of processing, data storage, and network resources reside in a single location. This often requires specialized rack and enclosure systems, appropriate heating, cooling, ventilation, security, fire suppression, and power distribution systems. The term may also refer to computing and data storage nodes in some contexts. Data centers can vary in scale between centralized or cloud data centers (the largest), regional data centers, and edge data centers (the smallest).

[0100] As used herein, references to a "layer" of an edge network can encompass various forms or types of edge networks and edge network configurations having common characteristics related to latency, timing, or distance, regardless of the use of terms such as "close edge," "local edge," "middle edge," "far edge," or a specifically named layer. Thus, references to a layer generally will not necessarily refer to a layer in the OSI model, but rather to any network portion or segment having a common tier or set of characteristics.

[0101] As used herein, the term "access edge layer" refers to the sub-layer of the infrastructure edge closest to end users or devices. For example, such a layer may be realized by edge data centers deployed at cellular network sites. The access edge layer may serve as the front line of the infrastructure edge and connect to an aggregation edge layer located higher in the hierarchy. Also, as used herein, the term "aggregation edge layer" refers to the layer of the infrastructure edge one hop away from the access edge layer. This layer may exist as a medium-sized data center in a single location or may be formed from multiple interconnected micro data centers, forming a hierarchical topology with the access edge, enabling greater collaboration, workload failover, and scalability than the access edge alone.

[0102] As used herein, the term "Network Functions Virtualization" (or NFV) refers to the migration of network functions from embedded services in dedicated equipment to software-based virtualized network functions (or VNFs) running on standard CPUs (e.g., standard x86® and ARM® servers, such as Intel® Xeon®, AMD® Epyc®, Opteron® processors, etc.) using industry-standard virtualization and cloud computing technologies. In some aspects, NFV processing and data storage will occur in edge data centers directly connected to local cellular sites within the infrastructure edge.

[0103] As used herein, the term "virtualized network function" (or VNF) refers to a software-based network function that runs on multi-function, multi-purpose computing resources (e.g., x86, ARM processing architectures) used by NFV in place of dedicated physical devices. In some aspects, some VNFs run in edge data centers at the infrastructure edge.

[0104] As used herein, the term "edge computing node" refers to a real-world, logical, or virtualized implementation of a computing-capable element in the form of a device, gateway, bridge, system, subsystem, or component, whether operating in server, client, endpoint, or peer mode, whether located at the "edge" of a network, or at a more connected location within the network. References to "node" as used herein are generally interchangeable with "device," "component," and "subsystem," while references to an "edge computing system" generally refer to a distributed architecture, configuration, or collection of multiple nodes and devices organized to achieve or provide some aspect of a service or resource in an edge computing setting.

[0105] As used herein, the term "cluster" refers to a set or group of entities as part of an edge computing system (or systems), in the form of physical entities (e.g., various computing systems, networks, or network groups), logical entities (e.g., applications, functions, security configurations, containers), etc. In some locations, a "cluster" is also referred to as a "group" or "domain." Cluster membership can change or be affected based on conditions or functions, including dynamic or property-based membership, network or system management scenarios, or various exemplary techniques described below that add, modify, or remove entities within a cluster. A cluster can also include or be associated with multiple layers, levels, or properties, including variations in security functions and results based on such layers, levels, or properties.

[0106] While many of the examples below are provided using specific cellular / mobile network terminology, including the use of 4G / 5G 3GPP network components (or anticipated terahertz-based 6G / 6G+ technologies), it will be understood that these examples may apply to many other deployments of wide-area and local wireless networks, and the integration of wired networks (including optical networks and associated fiber, transceivers, etc.). Where technically possible, the various network connections may be wired or wireless in any of the networks disclosed herein, and the resulting system may be a hybrid deployment of both wired and wireless network technologies. Furthermore, any of the wireless network connection standards described herein may be used to achieve the disclosed functionality, depending on the system or subsystem architecture requirements.

[0107] II. Specific examples of edge computing systems FIG. 1 is a block diagram 100 illustrating an overview of an edge computing architecture, including a layer of processing referred to in many of the following examples as the “edge cloud.” As shown, the edge cloud 110 is co-located at an edge location, such as an access point or base station 140, a local processing hub 150, or a central office 120, and thus may include multiple entities, devices, and equipment instances. The edge cloud 110 is located much closer to endpoint (consumer and producer) data sources 160 (e.g., autonomous vehicles 161, user equipment 162, business and industrial equipment 163, video capture devices 164, drones 165, smart city and building devices 166, sensors and IoT devices 167, etc.) than a cloud data center 130. The processing, memory, and storage resources provided at the edge of the edge cloud 110 are important for providing very low latency and response times for the services and functions used by the endpoint data sources 160 and reducing network backhaul traffic from the edge cloud 110 to the cloud data center 130, thus improving energy consumption and overall network utilization, among other benefits.

[0108] Processing, memory, and storage are scarce resources that generally decrease depending on the location of the edge (e.g., fewer processing resources are available at a consumer endpoint device than at a central office, or at a base station). However, the closer the edge location is to the endpoint (e.g., UE), the more space and power are often constrained. Thus, edge computing attempts to reduce the amount of resources required for network services by allocating more resources closer in network access time and geographically. In this way, edge computing attempts to bring computing resources to workload data or vice versa, as needed.

[0109] Below, we describe aspects of edge cloud architectures that cover multiple potential deployments and address constraints that some network operators or service providers may have in their own infrastructure. These include configuration variations based on edge location (e.g., because base-station-level edges may have more constrained performance and capacity in multi-tenant scenarios); configurations based on the type of processing, memory, storage, fabric, acceleration, or similar resources available to an edge location, location hierarchy, or location group; service, security, management, and orchestration capabilities; and related objectives for achieving end service availability and performance. These deployments can achieve processing at network layers that can be thought of as "near edge," "close edge," "local edge," "middle edge," or "far edge" layers, depending on latency, distance, and timing characteristics.

[0110] Edge computing is a development paradigm in which computations are performed at or near the "edge" of the network, typically through the use of computing platforms (e.g., x86 or ARM computing hardware architectures) implemented in base stations, gateways, network routers, or other devices reasonably close to the endpoint devices that generate and consume data (e.g., "local edge," "close edge," or "near edge"). For example, edge gateway servers may be equipped with a pool of memory and storage resources to perform real-time computations for low-latency use cases (e.g., autonomous driving or video surveillance) for connected client devices. Alternatively, as one example, base stations may be augmented with compute and acceleration resources to directly handle service workloads for connected user equipment without further communicating data over a backhaul network. Alternatively, as another example, central office network management hardware may be replaced with standardized compute hardware that runs virtualized network functions and provides computational resources for the execution of services and consumer functions for connected devices. Within edge computing networks, there may be scenarios within a service where compute resources are "moved" to data, and data is "moved" to compute resources. Alternatively, as an example, base station compute, acceleration, and network resources may be provided to service to scale to workload demands as needed by activating dormant capacity (subscription, capacity on demand, subscription) to manage corner cases, emergency situations, or to provide longevity for deployed resources over significantly longer implementation lifecycles.

[0111] FIG. 2 illustrates operational layers between endpoints, edge clouds, and cloud computing environments. Specifically, FIG. 2 illustrates an example compute use case 205 that utilizes edge cloud 110 among several exemplary layers of network computing. These layers begin with endpoint (device and thing) layer 200, which accesses edge cloud 110 to perform data production, analysis, and data consumption activities. Edge cloud 110 can span multiple network layers, such as edge device layer 210, which has gateways, on-premise servers, or network equipment (nodes 215) located in physically proximate edge systems; network access layer 220, which encompasses base stations, wireless processing units, network hubs, regional data centres, or local network equipment (devices 225); and any equipment, devices, or nodes located in between (in layer 212, not shown in detail). Network communication within the edge cloud 110 and between the various layers can occur via any number of wired or wireless mediums, including via the connectivity architectures and techniques described with reference to Figures 65-83.

[0112] Example latencies resulting from network communication distance and processing time constraints may range from less than milliseconds (ms) within the endpoint layer 200, to less than 5 ms at the edge device layer 210 (e.g., the "near edge" or "close edge" layer), to even 10-40 ms when communicating with nodes in the network access layer 220 (e.g., the "middle edge" layer). Outside the edge cloud 110 are the core network 230 and cloud data center 240 layers, each of which has increased latency (e.g., 50-60 ms at the core network layer 230 and 100 ms or more at the cloud data center layer, both of which may be considered "far edge" layers). As a result, operations with latencies of at least 50-100 ms or more at the core network data center 235 or cloud data center 245 will be unable to accomplish many time-critical functions of use case 205. Each of these latency values ​​is provided for purposes of illustration and comparison, and it will be understood that the use of other access network media and technologies may further reduce latency.

[0113] Various use cases 205 may access resources under heavy utilization from incoming streams due to multiple services utilizing the edge cloud. To achieve low-latency results, services running within the edge cloud 110 must balance various considerations in terms of (a) priority (throughput or latency) and quality of service (QoS) (e.g., autonomous vehicle traffic may have a higher priority than temperature sensors in terms of response time requirements; or performance sensitivity / bottlenecks may exist in compute / accelerator, memory, storage, or network resources, depending on the application); (b) reliability and resilience (e.g., depending on the application, some input streams may need to operate, traffic may be routed with mission-critical reliability, and some other input streams may be able to tolerate occasional failures); and (c) physical constraints (e.g., power, cooling, and form factor).

[0114] The end-to-end service view of these use cases includes the concept of service flow and is associated with transactions. Transactions detail the overall service terms for the entities consuming the service and related services in terms of resources, workload, workflow, business functionality, and business-level conditions. Services that execute under the described terms can be managed at each layer in a way that ensures real-time and runtime compliance with the transaction contract during the service lifecycle. If a component of a transaction fails to agree to an SLA, the overall system (the component of the transaction) can provide the ability to (1) understand the impact of the SLA violation, (2) augment other components in the system to restart the entire SLA transaction, and (3) implement remedial steps.

[0115] Thus, taking these variations and service capabilities into account, edge computing within edge cloud 110 can provide real-time or near-real-time service and response capabilities for multiple applications in use cases 205 (e.g., object tracking, video surveillance, connected cars, etc.), and can meet very low latency requirements for these multiple applications. These benefits enable entirely new classes of applications (e.g., VNFs, Function as a Service (FaaS), standard processes, etc.) that were previously unable to utilize cloud computing due to latency or other limitations.

[0116] However, along with the benefits of edge computing come the following caveats: Devices deployed at the edge are often resource-constrained, and therefore there is pressure on edge resource utilization. Typically, this is addressed by pooling memory and storage resources for use by multiple users (tenants) and devices. The edge may be power- and cooling-constrained, and therefore power usage must be accounted for by the most power-hungry applications. Many of these pooled memory resources are likely to use emerging memory technologies where more power requires more memory bandwidth, so there may be inherent power-performance tradeoffs in the terms of these pooled resources. Similarly, because edge locations may be unmanned and may even require authorized access (e.g., if housed at a third-party location), improved hardware and root-of-trust security are also required. These issues are magnified in edge clouds 110 in multi-tenant, multi-owner, or multi-access settings, especially when services and applications are required by many users, network utilization is dynamic, and the mix of multiple stakeholders, use cases, and services changes.

[0117] At a more general level, an edge computing system can be described as encompassing deployment of any number of the aforementioned layers (network layers 200-240) operating within the edge cloud 110 to provide collaboration from clients and distributed computing devices. For illustrative purposes, Figure 3 provides an abstracted overview of the distributed computing layers deployed in an edge computing environment. Various types of network links at the same (e.g., peer-to-peer) or different layers are also depicted.

[0118] FIG. 3 generally depicts an edge computing system 300 that provides edge services and applications to multi-stakeholder entities as distributed among one or more client computing nodes 302, one or more edge gateway nodes 312, one or more edge aggregation nodes 322, one or more core data centers 332, and a global network cloud 342 distributed across network layers. Implementations of edge computing system 300 can be provided at or on behalf of a telecommunications service provider (“telco” or “TSP”), an Internet of Things (IoT) service provider, a cloud service provider (CSP), an enterprise entity, or any number of other entities. Various implementations and configurations of system 300 may be dynamically provided, such as when tailored to meet service objectives.

[0119] Individual nodes or devices in the edge computing system 300 are located in specific layers, corresponding to layers 200, 210, 220, 230, and 240. For example, the client computing node 302 is located in the endpoint layer 200, while the edge gateway node 312 is located in the edge device layer 210 (local level) of the edge computing system 300. Furthermore, the edge aggregation node 322 (and / or the fog device 324, if deployed or operating with or within a fog networking configuration 326) is located in the network access layer 220 (intermediate level). Fog computing (or "fogging") generally refers to the extension of cloud computing to the edge of an enterprise's network or the ability to manage transactions across a cloud / edge landscape, typically in a federated, distributed, or multi-node network. Some forms of fog computing provide an alternative to cloud computing by deploying computing, storage, and networking services between end devices and cloud computing data centers. Some forms of fog computing also provide the ability to manage workload / workflow level services with respect to the overall transaction by pushing specific workloads to the edge or to the cloud based on their ability to meet overall service level agreements.

[0120] Fog computing in many scenarios provides a decentralized architecture and acts as an extension to cloud computing by collaborating with one or more edge node devices to provide localized control, configuration, and management for end devices. Furthermore, fog computing provides edge resources the ability to identify similar resources and collaborate to create edge-local clouds that can be used alone or in combination with cloud computing to complete computing, storage, or connectivity-related services. Fog computing also enables cloud-based services to extend their reach to the edge of a network of devices, providing local, rapid access to edge devices. Thus, some forms of fog computing provide operations consistent with edge computing as discussed herein; aspects of edge computing discussed herein are also applicable to fog networks, fogging, and fog configurations. Furthermore, aspects of the edge computing system described herein may be configured as fog, or fog aspects may be integrated into an edge computing architecture.

[0121] The core data center 332 is located at the core network layer 230 (at a regional or geographically centralized level), while the global network cloud 342 is located at the cloud data center layer 240 (at the global or worldwide layer). The use of "core" is provided as a term for a centralized network location deeper within the network that is accessible by multiple edge nodes or components, but "core" does not necessarily indicate the "center" or deepest location of the network. Thus, the core data center 332 may be located within, at, or near the edge cloud 110. While FIG. 3 shows an exemplary number of client computing nodes 302, edge gateway nodes 312, edge aggregation nodes 322, edge-core data centers 332, and global network clouds 342, it should be understood that the edge computing system 300 may include additional devices or systems at each layer. Devices at any layer can be configured as peer nodes to each other and thus can operate in a cooperative manner to meet service goals. Furthermore, as shown in Figure 3, the number of components in each layer 200, 210, 220, 230, 240 generally increases at each lower level (e.g., as you move closer to the endpoints). Thus, one edge gateway node 312 can serve multiple client computing nodes 302, and one edge aggregation node 322 can serve multiple edge gateway nodes 312.

[0122] Consistent with the examples provided herein, the client computing node 302 may be embodied as any type of endpoint component, device, appliance, or other entity capable of communicating as a producer or consumer of data. Furthermore, the labels "node" or "device" used in the edge computing system 300 do not necessarily imply that such node or device operates in a client or slave role; rather, any node or device in the edge computing system 300 may refer to an individual entity, node, or subsystem, including separate or connected hardware or software configurations for facilitating or utilizing the edge cloud 110.

[0123] Thus, edge cloud 110 is formed from the network components and functional features operated by and within edge gateway node 312 and edge aggregation node 322 of layers 210, 220, respectively. Edge cloud 110 can be embodied as any type of network providing edge computing and / or storage resources located in proximity to radio access network (RAN)-enabled endpoint devices (e.g., mobile computing devices, IoT devices, smart devices, etc.), shown in FIG. 3 as client computing nodes 302. In other words, edge cloud 110 may be envisioned as an “edge” connecting traditional network access points and endpoint devices that serve as entry points into service provider core networks, including mobile carrier networks (e.g., Global System for Mobile Communications (GSM) networks, Long Term Evolution (LTE) networks, 5G / 6G networks, etc.), while providing storage and / or computing capabilities. Other types and forms of network access (e.g., Wi-Fi networks, long-range wireless networks, wired networks including optical networks) may also be used in place of or in combination with such 3GPP carrier networks.

[0124] In some examples, the edge cloud 110 may form part of or provide a point of entry into or across a fog networking configuration 326 (e.g., a network of fog devices 324, not shown in detail), which may be embodied as a system-level horizontal and distributed architecture that distributes resources and services to perform specific functions. For example, a federated distributed network of fog devices 324 may perform aspects of computing, storage, control, or networking in the context of an IoT system configuration. Other networked, aggregated, and distributed functions may exist in the edge cloud 110 between the core data center 332 and client endpoints (e.g., client computing nodes 302). Some of these are discussed in the following section in the context of network function or service virtualization, including the use of virtual edges and virtual services coordinated for multiple stakeholders.

[0125] As described in further detail below, the edge gateway nodes 312 and edge aggregation nodes 322 cooperate to provide various edge services and security to the client computing nodes 302. Furthermore, the client computing nodes 302 may be fixed or mobile, and each edge gateway node 312 may cooperate with other edge gateway devices to propagate currently offered edge services, associated service data, and security as the corresponding client computing node 302 moves around an area. To do so, the edge gateway nodes 312 and / or edge aggregation nodes 322 may support multi-tenant and multi-stakeholder configurations, and services from multiple service providers, owners, and multiple consumers may be supported and coordinated via a single or multiple computing devices.

[0126] Various security approaches may be utilized within the edge cloud 110 architecture. In a multi-stakeholder environment, there may be multiple loadable security modules (LSMs) used to provide policies that enforce stakeholder interests. An enforcement point environment may support multiple LSMs that apply a combination of loaded LSM policies (e.g., the most restrictive policy may apply, such as if stakeholder A, B, or C restricts access, then access is restricted). Within the edge cloud 110, each edge entity may provide an LSM that enforces the edge entity's interests. A cloud entity may provide an LSM that enforces the cloud entity's interests. Similarly, various fog and IoT network entities may provide LSMs that enforce the fog entity's interests.

[0127] In these examples, a service can be considered in terms of a series of contracts or transactions performed against components, whether considered at the component level or at a human-perceptible level. Thus, a user who has a service contract with a service provider expects the service to be provided under the terms of the SLA. While not discussed in detail, the use of edge computing technologies discussed herein can play a role during contract negotiation and evaluation of contract performance (to identify which components are required by the system to perform the service, how the system responds to service conditions and changes, etc.).

[0128] "Services" is a broad term that is often applied in a variety of contexts, but generally refers to a relationship between two entities in which one entity provides and performs work for the benefit of the other. However, services provided by one entity to another must be performed with specific guidelines that ensure trust between the entities and govern the transaction in accordance with the terms and conditions of a defined contract at the initiation, during, and completion of the service.

[0129] An example relationship between services used in an edge computing system is depicted in Figure 4. In an edge computing scenario, there are several interdependent services and transaction layers 400 in operation; these services form a "service chain." At the lowest level, components comprise a system. These systems (or resources) communicate and collaborate with each other to provide multiple services to each other and to other persistent or transient entities in the environment. These entities can then provide services that can be consumed by humans. At this layer, the services provided at each layer must be transactionally connected to ensure that the individual components (or sub-entities) providing the services adhere to contractually agreed-upon objectives and specifications. Deviations at each layer can have a global impact on the entire service chain.

[0130] One type of service that may be offered in the hierarchy shown in Figure 4 is silicon-level services. For example, Software Defined Silicon (SDSi) type hardware provides the ability to guarantee low-level compliance to transactions through the ability to intrascale, manage, and guarantee the delivery of operational service level agreements. The use of SDSi and similar hardware controls provides the ability to associate functions and resources within a system with specific tenants and manage their entitlements to those resources. The use of such capabilities allows computing resources to be dynamically "brought in" to workloads.

[0131] For example, in the case of SDSi, operational level agreements can define "transaction throughput" or "timeliness," and systems (or resources) can sign up to guarantee specific service level specifications (SLS 430) and service level agreement (SLA 410) targets (SLO 420). SDSi hardware also empowers infrastructure and resource owners to access and manage (add / remove) product features and authorize silicon components (e.g., components of the configuration system 442 that generate metric telemetry 440) to freely scale hardware capabilities and utilization up or down. It also provides the ability to perform deterministic feature assignment on a per-tenant basis. It also provides the ability to combine deterministic orchestration and service management with dynamic (or subscription-based) activation of features without the need to interrupt running services, client operations, or reset or reboot the system.

[0132] At the lowest layer, SDSi can provide services and guarantees for systems to ensure proactive compliance with contractually agreed-upon service level specifications that individual resources within the system must deliver. Furthermore, SDSi provides the ability to manage one or more tenants' contractual rights (entitlements), usage, and associated financials per component, as well as silicon-level features (e.g., SKU features). Silicon-level features can be associated with compute, storage, or network capabilities, performance, determinism, or even features such as security, encryption, and acceleration. These capabilities not only enable tenants to achieve specific service level agreements, but also assist with management and data collection, ensuring transactional and contractual agreements at the lowest manageable component level.

[0133] A higher layer in the services hierarchy, resource level services, includes systems that provide the capability (fully or through configuration) to meet workload demands by obtaining and enabling system level functionality via SDSi or through the configuration of individually addressable resources (compute, storage, and network).

[0134] Workflow-level services, which are higher layers in the service hierarchy, are horizontal because service chains may have workflow-level requirements. Workflows describe the dependencies between workloads to provide specific service-level objectives and requirements for end-to-end services. These services may include features and capabilities such as high availability, redundancy, resiliency, fault tolerance, and load balancing. Workflow services define the dependencies and relationships between resources and systems to ensure end-to-end services, describe associated network and storage requirements, and also describe transaction-level requirements and associated contracts. Workflow-level services are typically measured against service-level objectives and have required and expected service requirements.

[0135] At higher levels in the service hierarchy, Business Function Services (BFSs) operate. These services are various elements of a service that have relationships with each other and provide specific functionality to customers. In the edge computing and autonomous driving example, a business function might comprise a service such as "timely arrival at an event." This service requires the coordination of several business functions to achieve a user entity's goal. These business functions include: GPS guidance, RSU (Roadside Unit) awareness of local traffic conditions, the user entity's payment history, and the user entity's authorization of resources. Furthermore, when these BFSs provide services to multiple entities, each BFS manages its own SLA and is aware of its ability to address demands on its resources (workload and workflow). As requirements and demands increase, service change requirements are communicated to service entities at the workflow and resource levels, so that the service entities can provide insight into their own capabilities for fulfillment. This step supports the overall transaction and service delivery to the next layer.

[0136] Business Level Services (BLS), the highest layer of service in the service hierarchy, are tied to the functionality provided. At this level, the customer or entity may not know how the service is configured or what components are used, managed, and tracked to deliver the service. The primary objective of Business Level Services is to achieve the goals set by the customer in accordance with the overall contractual terms and conditions established between the customer and provider in the agreed-upon financial contract. BLS consists of several Business Functional Services (BFS) and an overall SLA.

[0137] This and other service management capabilities described herein are designed to meet the diverse needs of edge computing, with its inherently complex resource and service interactions. This service management architecture is intended to inherently address some of the resource basic services within the framework, rather than relying on agents or middleware functions. Services such as locate, discover, address, trace, track, identify, register, etc., can be located virtually adjacent to the resource as it appears in the framework, and resource or security domain managers or owners can use management rules and policies to ensure the orderly discovery, registration, and authentication of resources.

[0138] Additionally, any number of edge computing architectures described herein can be adapted with service management functions. These functions enable the system to constantly recognize and record information about resource movement, vectors, and direction, and to fully describe these characteristics as both telemetry and metadata associated with the device. These service management functions can be used for resource management, billing, and metering, as well as security elements. Similar functions also apply to associated resources, where unintelligent devices such as sensors may be connected to more manageable resources such as edge gateways. The service management framework is aware of changes in resource storage or encapsulation. Because nodes and components may be directly accessible or indirectly managed through parent devices or devices with alternative responsibilities, either for short periods of time or throughout their lifecycle, this type of structure is relayed to the service framework through its interfaces and made available to external query mechanisms.

[0139] Furthermore, this service management framework is service-aware, naturally balancing service delivery requirements with the availability and capacity of access and resources for data uploads to the data analysis system. If network transport degrades, fails, or shifts to higher cost or lower bandwidth capabilities, service policy monitoring capabilities provide alternative analysis and service delivery mechanisms within user privacy or cost constraints. Using these capabilities, policies can trigger invocation of analysis and dashboard services at the edge, ensuring continuous service availability at reduced fidelity or granularity. Once network transport is re-established, regular data collection, upload, and analysis services can be resumed.

[0140] A. Edge Computing Configuration and Deployment Multi-stakeholder edge computing system deployments can be arranged and coordinated to enable the deployment of multiple services and virtual edge instances among multiple edge nodes and subsystems for use by multiple tenants and service providers. In an example system applicable to a cloud service provider (CSP), edge computing system deployments can be provided via an "over-the-top" approach to deploy edge computing nodes as a supplementary tool to cloud computing. In a contrasting example system applicable to a communications service provider (TSP), edge computing system deployments can be provided via a "network aggregation" approach to deploy edge computing nodes where network access (from different types of data access networks) is aggregated. Figures 5 and 6 contrast these over-the-top and network aggregation approaches for networking and services in each edge computing system. However, these over-the-top and network aggregation approaches can be implemented together in hybrid or merged approaches or configurations, as suggested in later examples.

[0141] In FIG. 5, various client endpoints 510 (in the form of mobile devices, computers, autonomous vehicles, business computing equipment, industrial processing equipment) provide requests 520 for services or data transactions and receive responses 530 for services or data transactions from the edge cloud 110 (e.g., via wireless or wired networks 540). Within the edge cloud 110, a CSP can deploy various computing and storage resources, such as edge content nodes 550, to provide cached content from a distributed content delivery network. Other available computing and storage resources on the edge content nodes 550 can be used to run other services and fulfill other workloads. The edge content nodes 550 and other systems in the edge cloud 110 are connected to a cloud or data center 570 and use a backhaul network 560 to meet the higher latency requirements from the cloud / data center for websites, applications, database servers, etc.

[0142] In FIG. 6 , various client endpoints 610 (in the form of mobile devices, computers, autonomous vehicles, business computing devices, and industrial processing devices) exchange requests and responses specific to the type of endpoint network aggregation. For example, computers, business computing devices, and industrial processing devices can gain network access over a wired broadband network by exchanging requests and responses 622 over an on-premises network system 632. Mobile computing devices can gain network access over a wireless broadband network by exchanging requests and responses 624 over a cellular network tower 634. Autonomous vehicles can gain network access for requests and responses 626 over a wireless vehicular network through a street-based network system 636. However, regardless of the type of network access, the TSP can deploy aggregation points 642, 644 within the edge cloud 110 to aggregate traffic and requests. Accordingly, within the edge cloud 110, the TSP can deploy various computing and storage resources, such as edge aggregation nodes 640, to provide requested content. The edge aggregation nodes 640 and other systems in the edge cloud 110 are connected to a cloud or data center 660, which uses a backhaul network 650 to meet higher latency requirements from the cloud / data center for websites, applications, database servers, etc. (Additional or integrated instances of the edge aggregation nodes 640 and aggregation points 642, 644 may also exist within the edge cloud 110 or other areas of the TSP infrastructure, including those deployed on a single server framework.)

[0143] As an extension of either the CSP or TSP configuration, FIGS. 7 and 8 illustrate the deployment and orchestration of a virtual edge configuration across multiple edge nodes and an edge computing system operating among multiple tenants. Specifically, FIG. 7 illustrates the coordination of a first edge node 722 and a second edge node 724 in an edge computing system 700 to fulfill requests and responses for various client endpoints 710 (e.g., smart city / building systems, mobile devices, computing devices, business / logistics systems, industrial systems, etc.) accessing various virtual edge instances. Similar to the examples of FIGS. 5 and 6, the virtual edge instances provide edge computing capacity and processing in an edge cloud along with access to a cloud / data center 740 for higher latency requirements for websites, applications, database servers, etc. However, the edge cloud enables the coordination of processing among multiple edge nodes for multiple tenants or entities.

[0144] In the example of FIG. 7 , these virtual edge instances include a first virtual edge 732 provided to a first tenant (tenant 1) that provides a first combination of edge storage, computing, and services, and a second virtual edge 734 that provides a second combination of edge storage, computing, and services. The virtual edge instances 732, 734 may be distributed among edge nodes 722, 724, including scenarios where requests and responses are fulfilled by the same or different edge nodes. The configuration of the edge nodes 722, 724 operates in a distributed yet coordinated fashion based on the edge provisioning function 750. The ability of the edge nodes 722, 724 to provide coordinated operation for applications and services across multiple tenants occurs based on the orchestration function 760.

[0145] It should be understood that some of the devices in 710 are multi-tenant devices, with Tenant 1 functioning within Tenant 1's "slice" while Tenant 2 functions within Tenant 2's slice. (In further examples, there may be additional or sub-tenants, and each tenant may even be specifically granted tradable privileges tied to a specific set of capabilities, down to entirely specific hardware functionality.) A trusted multi-tenant device may further include tenant-specific cryptographic keys, such that the combination of key and slice can be considered a "Root of Trust" (RoT) or tenant-specific RoT. Furthermore, a single DICE hardware building block may be dynamically configured using a Device Identity Composition Engine (DICE) architecture, such that it can be used to build a layered trusted computing infrastructure for layering device capabilities (such as field programmable gate arrays (FPGAs)). The RoT can also be used for trusted computing contexts, enabling "fan-out," useful for supporting multi-tenancy. Within a multi-tenant environment, each edge node 722, 724 can act as an LSM or security enforcement point for local resources allocated to multiple tenants per node. Additionally, tenant runtimes and application executions (e.g., instances 732, 734) can serve as LSMs or other security enforcement points, creating a virtual edge abstraction of resources potentially spanning multiple physical hosting platforms. Finally, the orchestration function 760 in the orchestration entity can act as an LSM or security enforcement point for marshaling resources along tenant boundaries.

[0146] Edge computing nodes can partition resources (memory, CPU, GPU, interrupt controller, I / O controller, memory controller, bus controller, etc.), and each partition may include a RoT function. Furthermore, the DICE model of fan-out and tiering may be applied to edge nodes. Cloud computing nodes, consisting of containers, FaaS engines, servlets, servers, or other compute abstractions, can be partitioned according to the DICE tiering and fan-out structure and support their own RoT contexts. Therefore, individual RoT spanning devices 710, 722, and 740 can cooperate to establish a distributed trusted computing foundation (DTCB) that can establish tenant-specific virtual trusted secure channels linking all elements end-to-end.

[0147] In the example of FIG. 8, the edge computing system 800 is extended to provide orchestration of multiple applications through the use of containers (self-contained, deployable units of software that provide code and necessary dependencies) in a multi-owner, multi-tenant environment. The multi-tenant orchestrator can be used to perform key management, trust anchor management, and other security functions related to the provisioning and lifecycle of the trusted "slice" concept of FIG. 7. The orchestrator can use the DICE layering and fan-out structure to create a tenant-specific origin of trust context. Thus, the orchestration function 840 provided by the orchestrator, described below, can participate as a tenant-specific orchestration provider.

[0148] Similar to the scenario of FIG. 7, the edge computing system 800 is configured to satisfy requests and responses to various client endpoints 810 from multiple virtual edge instances (and from cloud or remote data centers, not shown). The use of these virtual edge instances simultaneously supports multiple tenants and multiple applications (e.g., augmented reality (AR) / virtual reality (VR), enterprise applications, content delivery, gaming, and compute offloading). Furthermore, within a virtual edge instance, multiple types of applications (e.g., normal applications, latency-sensitive applications, latency-critical applications, user-plane applications, networking applications, etc.) may exist. Additionally, the virtual edge instances may also span multiple owners' systems (or individual computing systems and resources shared or jointly managed by multiple owners) in different geographic locations.

[0149] Within the edge cloud, a first edge node 820 (operated by a first owner) and a second edge node 830 (operated by a second owner) each operate an orchestrator to coordinate the execution of various applications within the virtual edge instances provided to individual tenants. The edge nodes 820 and 830 are coordinated based on an edge provisioning function 850, while the operation of various applications is coordinated by an orchestration function 840. Furthermore, the orchestrator can identify specific hardware features provided to one owner (but hidden from a second owner) to ensure that services are completed according to their respective SLAs, even if provided across ownership boundaries. Thus, the virtual edge, container orchestrator, and service / app orchestrator can provide LSM or other security enforcement points for node-specific resources bound to specific tenants.

[0150] 9 illustrates various computing configurations for deploying containers in an edge computing system. As simplified examples, system configurations 910 and 920 illustrate configurations in which a container manager (e.g., container managers 911, 921, and 931) is adapted to launch containerized pods, functions, and Function-as-a-Service (FaaS) instances via compute nodes (915 in configuration 910) or separately run containerized virtualized network functions via compute nodes (923 in configuration 920). This configuration is adapted for use with multiple tenants in a system configuration 930 (using compute nodes 936), where containerized pods (e.g., pod 912), functions (e.g., function 913, VNFs 922, 936), and Function-as-a-Service instances (e.g., FaaS instance 915) are launched within virtual machines (e.g., VMs 934, 935 for tenants 932, 933) specific to individual tenants (apart from running virtualized network functions). This configuration is further adapted for use in a system configuration 940 that provides for the execution of various functions, applications, and features in containers 942, 943 or compute nodes 944, coordinated by a container-based orchestration system 941.

[0151] The system configurations shown in Figures 8 and 9 provide an architecture that treats VMs, containers, and functions equally from an application composition perspective (and the resulting application is a combination of these three components). Each component can include the use of one or more accelerator (FPGA, ASIC) components as local backends. In this way, an application can be partitioned across multiple edge owners that are orchestrated by an orchestrator.

[0152] In the context of Figure 9, the container manager, container orchestrator, and individual nodes can provide LSMs or other security enforcement points. However, in either of the configurations of Figures 8 and 9, tenant isolation may be coordinated, with resources allocated to one tenant being different from those allocated to a second tenant, but the edge owner collaborates to ensure resource allocations are not shared across tenant boundaries. Tenants can also grant "usage" on a subscription, transactional, or contractual basis, allowing resource allocations to be isolated across tenant boundaries. In these contexts, virtualization, containerization, enclaves, and hardware partitioning methods may be used by the edge owner to enforce tenant isolation. Other isolation environments may include bare metal (dedicated) devices, virtual machines, containers, virtual machines on containers, or a combination thereof. Functionality, such as functionality provided in a FaaS environment, described further below, can be executed in any of these isolation environments to enforce tenant boundaries.

[0153] 10A and 10B illustrate a distributed edge computing deployment using federated computing functions and services across edge computing system instances. Within the edge computing environment 1010, multiple geographically distributed edges (e.g., 1011, 1013, 1014) include a container manager (e.g., 1012) for coordinating the execution of services through applications (e.g., 1015) and functions (e.g., 1016) across computing resources (e.g., thin edge instance 1013 or client PC 1014) on the distributed edges. Within the edge computing environment 1020, a distributed edge (thin edge instance 1021) collaborates with other distributed edges (large / medium edge instance 1030) that have additional processing power. For example, an application or function (e.g., 1022 or 1023) running on a particular distributed edge instance (thin edge 1021) can further invoke GPU processing power within the edge cloud (provided by large / medium edge instance 1030 in the form of GPU-as-a-Service 1030); or, as another example, an application or function (e.g., 1025, 1026) on a client computer (client PC 1024) can further invoke processing power within the edge cloud (provided by large / medium edge instance 1030 in encryption-as-a-service 1035). Other applications, functions, functions-as-a-service, or accelerators-as-a-service (e.g., 1031, 1032, 1033, 1034) can be provided by the edge cloud (e.g., compute 1036) and may be coordinated, distributed, etc. across edge nodes. Furthermore, in an edge computing architecture, each edge node, VM, container, or even bare metal node may be self-describing, self-aware, and self-managing, creating a set of presentations that other nodes can use in an "as-a-service" model.

[0154] In further examples, software-defined or controlled silicon hardware and other configurable hardware aspects may be integrated with the applications, features, and services of Figures 10A and 10B, as well as other scenarios discussed herein. Software-defined silicon may be used to guarantee the ability of any resource or hardware component to meet a contract or service level agreement based on the component's ability to correct a workload or portion thereof (e.g., by upgrading, reconfiguring, or providing new features within the hardware configuration itself).

[0155] FIG. 11 illustrates edge computing configurations linked together in an edge mesh to provide various forms of edge computing systems. The simplest example configuration 1110 illustrates an edge mesh. A more advanced configuration 1120 illustrates sidecar loading (sidecars denoted as "SC") used to operate aspects of the edge mesh along with microservices (with microservices denoted as "MS"). The more advanced configuration illustrates a first edge mesh 1130A connected to a second edge mesh 1130B to provide edge-to-edge connectivity. In these environments, the microservices provided by the edge computing nodes may be treated as functions, while the sidecars are functions that support connectivity with other functions.

[0156] A valuable feature of sidecars is that they provide a point of enforcement for LSMs and other security policies, and their environments have a "trusted path" relationship with the paired container environment. Sidecars can share data and state. Sidecars do not have to be "trusted" or "reliable" to the same extent that secure enclaves are recognized as trusted execution environments; however, sidecars are assumed to be at least as trusted as their paired containers. Furthermore, sidecars are often used to mediate interactions with external entities, providing a sandbox environment where different staging and filtering can be applied. This provides functionality similar to an application firewall that is specific to the paired container.

[0157] Thus, a sidecar can provide a trusted execution environment to support security functions such as cryptographic key generation, storage, and usage. Sidecars can also enable security-sensitive computations that protect privacy, intellectual property, content, or other information assets from less hardened mesh nodes. Furthermore, a sidecar with trusted computing capabilities can attest to microservice configurations to peer microservices and sidecar nodes. Nesting of microservices / sidecar mesh nodes can attest to the nesting or mesh structure, such that configurations of microservices and sidecar nodes can be evaluated for correct / incorrect structure, connections, and topology.

[0158] The mesh architecture provided by configurations 1110, 1120 allows for a network (cascade) of functions to exist. For example, a complex program may consist of a top-level "inner loop," which may in turn consist of several inner-inner loops, which in turn consist of inner-inner-inner loops, and so on. Each nesting of inner loops can be supported by accelerator offloading. Thus, many complex or connected scenarios can be enabled by these edge computing configurations.

[0159] It should be understood that the edge computing systems and configurations described herein are applicable to a variety of solutions, services, and / or use cases. As an example, FIG. 12 illustrates a simplified vehicular computing and communications use case involving mobile access to applications in an edge computing system 1200 that implements the edge cloud 110. In this use case, individual client computing nodes 1210 may be embodied as in-vehicle computing systems (e.g., in-vehicle navigation and / or infotainment systems) located within corresponding vehicles that communicate with an edge gateway node 1220 while traveling on a road. For example, the edge gateway node 1220 may be located within a roadside cabinet or other enclosure integrated into another, separate, mechanical utility structure that may be located along a road, at a road intersection, or at other locations near a road. As each vehicle moves along the road, the connection between its client computing node 1210 and a particular edge gateway device 1220 can be propagated to maintain a consistent connection and context for the client computing node 1210. Similarly, mobile edge nodes can aggregate for high-priority services or (as in the case of drones) according to throughput or latency requirements for basic services. Each edge gateway device 1220 includes some amount of processing and storage capability, so that some processing and / or storage of data for a client computing node 1210 may be performed on one or more edge gateway devices 1220.

[0160] The edge gateway device 1220 may communicate with one or more edge resource nodes 1240, illustratively embodied as computing servers, equipment, or components located at or within a communications base station 1242 (e.g., a base station of a cellular network). As described above, each edge resource node 1240 includes a certain amount of processing and storage capability, and thus, some processing and / or storage of data for the client computing node 1210 may be performed at the edge resource node 1240. For example, processing of less urgent or important data may be performed by the edge resource node 1240, while processing of more urgent or important data may be performed by the edge gateway device 1220 (e.g., depending on the capabilities of each component or information in the request indicating the urgency or importance). If processing priority changes during processing activity based on data access, data location, or latency, the work may continue on the edge resource node. Similarly, configurable systems or hardware resources themselves (e.g., via a local orchestrator) can be activated to provide additional resources to meet new demands (e.g., adapting computational resources to workload data).

[0161] The edge resource nodes 1240 also communicate with a core data center 1250, which may include servers, equipment, and / or other components located in a central location (e.g., a central office of a cellular communication network). The core data center 1250 may provide a gateway to a global network cloud 1260 (e.g., the Internet) for the operation of the edge cloud 110 formed by the edge resource nodes 1240 and the edge gateway devices 1220. Additionally, in some examples, the core data center 1250 may include some processing and storage capacity, and some processing and / or storage of data for client computing devices (e.g., less urgent or critical, or more complex processing) may be performed in the core data center 1250.

[0162] The edge gateway node 1220 or edge resource node 1240 may provide stateful application 1232 and use of a geographically distributed database 1234. While the application 1232 and database 1234 are illustrated as horizontally distributed at the edge cloud layer, it will be understood that application resources, services, or other components may be vertically distributed throughout the edge cloud (including portions of the application running on client computing nodes 1210 and other portions running on edge gateway nodes 1220 or edge resource nodes 1240). Furthermore, as previously described, peer relationships may exist at any level to fulfill service objectives and obligations. Furthermore, data for a particular client or application may move from edge to edge based on changing conditions (e.g., based on the availability of acceleration resources, following vehicle movement, etc.). For example, predictions can be made to identify future owners based on the “decay rate” of access, or when data or computational access will no longer be feasible. These and other services can be utilized to complete the tasks required to keep transactions compliant and loss-free.

[0163] In a further example, FIG. 12 can utilize various types of mobile edge nodes, such as edge nodes hosted in vehicles (cars / trucks / trams / trains) or other mobile units, where the edge nodes move to other geographic locations along with the platform hosting them. In vehicle-to-vehicle communications, individual vehicles may even act as network edge nodes for other vehicles (e.g., to cache, report, aggregate data, etc.). Thus, it will be appreciated that application components provided at various edge nodes may be distributed in static or mobile configurations, including coordination between some functions or operations at individual endpoint devices or edge gateway nodes 1220, some other functions at edge resource nodes 1240, and other functions at core data centers 1250 or global network clouds 1260.

[0164] In a further configuration, an edge computing system can implement FaaS computing functionality by using respective executable applications and functions. In one example, a developer writes function code (e.g., "computer code" herein) representing one or more computer functions, and the function code is uploaded to a FaaS platform, provided by, for example, an edge node or a data center. A trigger, such as a service use case or an edge processing event, initiates execution of the function code using the FaaS platform.

[0165] In FaaS examples, containers are used to provide the environment in which function code executes. A container can be any isolated execution entity, such as a process, a Docker or Kubernetes container, or a virtual machine. Within edge computing systems, various data center, edge, and endpoint (including mobile) devices are used to "spin up" (activate and / or allocate function actions) functions that scale on demand. Function code executes in physical infrastructure (e.g., edge computing nodes) devices and the underlying virtualized containers. Finally, containers are "spun down" (deactivated and / or deallocated) in the infrastructure upon completion of execution.

[0166] Further aspects of FaaS can enable the deployment of edge functions as a service, including support for individual functions in support of Edge Computing as a Service (Edge-as-a-Service or "EaaS"). Additional features of FaaS can include: a granular billing component that allows customers (e.g., computer code developers) to pay only when their code is executed; common data storage that stores data for reuse by one or more functions; orchestration and management among individual functions; function execution management, parallelization, and integration; management of container and function memory space; coordination of acceleration resources available to functions; and distribution of functions among containers (including "warm" containers that are already deployed or running, as well as "cold" containers that require initialization, deployment, or configuration).

[0167] In a further configuration, aspects of orchestration can be implemented in edge computing systems through the service aspects of an "orchestration-as-a-service (OaaS)" deployment, enabling distribution of stakeholders among the many aspects of edge orchestration and multi-tenancy. In one example, tenants of an edge computing system discover their OaaS provider as part of the SLA creation process (operating as part of a bootstrap function, configuration wizard, storefront, etc.). The technical capabilities necessary to support this discovery and usage may be baked into each device by the manufacturer, or an "onboarding"-type procedure may occur with each OaaS provider that tenants select and use within the edge computing system. Furthermore, during the SLA creation process, the OaaS provider can isolate what resources, conditions, or features are required from those available from the pool and create individual service requests for enablement / activation or subscription to specific features / functionality to utilize the resources.

[0168] Various types of hardware modifications and configurations can be implemented within edge computing devices to support OaaS. For example, hardware capabilities may provide an OaaS trust anchor upfront or provide information to enable the hardware manufacturer to act as a trusted clearinghouse for brokering OaaS deployments. Other types of software and service modifications and configurations proposed herein may also be implemented within edge computing systems to support OaaS features.

[0169] B. Workload Distribution and Management Figure 13 compares operational considerations for edge computing across operational network layers. Specifically, Figure 13 illustrates the degree and type of tradeoffs experienced between different computing processing locations. Various service characteristics 1302-1318 are increased or decreased at various layers of the network (e.g., details of devices located between the network access layer 220 and core network layer 230 in Figure 2), such as between an edge gateway 1320 (e.g., a base station, access point) and an edge cabinet 1330 (e.g., a street or roadside cabinet, on-premise cabinet) in a proximity device edge cloud, a point of aggregation in a central office 1340, a point of aggregation in a data center 1350, or a cloud data center for Internet or over-the-top services 1360. As one moves closer to the edge endpoint, one may experience increases in enabling new use cases 1302 (e.g., distributed and contextualized AI, real-time FaaS), reduced latency 1304 (e.g., reduced response time, faster user experience), increased throughput 1306 (e.g., federated allocation, load balancing, etc.), and reduced total cost of ownership and traffic (e.g., due to improved endpoint or edge filtering, caching, etc.). However, as one moves closer to the cloud data center, one may see improvements in security 1310 (e.g., due to improved isolation, data protection), reduced physical constraints 1312 (e.g., power, space, security, and thermal constraints), reduced management and maintenance 1314, the ability to reuse wired and wireless infrastructure 1316, and more complex orchestration 1318.

[0170] These trade-offs are magnified by mobility characteristics and different types of communication networks. Figure 14 illustrates operational deployments and latencies for edge computing between operational network layers in a cellular (mobile) wireless network. Specifically, Figure 14 illustrates an example of network latencies experienced at each of the mobile network layers 1420-1470 in an exemplary national deployment, ranging from the local layer 1420 (e.g., typically less than 1 ms to a small cell base station or access point, which may be considered the "local" or "close edge" layer), to the on-premise layer 1430 (e.g., with on-premise equipment or cloudlets / edgelets, which are also typically less than 1 ms, and may be considered the "enterprise edge," "local edge," or "close edge" layer), to the base station layer 1440 (e.g., with cell tower base stations or other wireless network gateways, which are typically between 1-5 ms, and may be considered the "near edge" layer). The figures provide examples of round-trip timings for the following layers: the aggregation layer 1450 (e.g., having multiple aggregation servers or locations, typically in the range of 1-2 ms per 100 kilometers (KM) plus 5 ms), the core network layer 1460 (e.g., having multiple core network servers or locations, typically in the range of 5 ms per 400 KM plus 5 ms, based on the currently used telecommunications technology, which may be considered the "far edge" layer), and the cloud layer 1470 (e.g., having one or more cloud processing locations, typically greater than 60 ms). Such latencies are provided for illustrative purposes only and may vary depending on the type of communications technology involved (but are limited by the speed of light). Edge-cloud system deployments (e.g., implementations of the edge cloud 110) are discussed below in connection with layers 1420-1450.

[0171] Endpoint devices used by end users or accessible through the nearby Local Layer 1420 may be considered "far edge" devices. Devices in this scenario are likely to offer the lowest "latency." However, at some point, the far edge device may become computationally limited or power inefficient when required to perform a given task. For example, at some point in network traffic loads, AR / VR use cases experience severe degradation (even to the point of offering worse performance than running the workload solely at the far edge of the device itself).

[0172] On-premise computing at the on-premise layer 1430 (sometimes known as the enterprise edge layer) is the next potential tier in a low-latency network edge architecture. On-premise refers to a location (typically on customer premises) capable of hosting a certain amount of compute (from a small form factor rack to multiple racks). Such on-premise computing platforms may be owned and operated by an enterprise, a cloud service provider, or a communications service provider.

[0173] Computing at base stations within the base station layer 1440 (sometimes known as the near or close edge layer) can aggregate multiple antennas and often be hosted as the first potential edge from a communications service provider's perspective. Base stations may run virtual radio access network (e.g., vRAN)-type workloads to handle 5G wireless traffic. The primary design challenges for running other services at base stations relate to (1) limited space, (2) physical exposure requiring more security and better thermal countermeasures, (3) limited power budget, and (4) the total cost of ownership (TCO) or operating expenses (OPEX) that arise from managing such a highly distributed computing environment. While we have outlined the challenges of deploying services at base stations, it is important to emphasize that base stations are still one of the unique points in the infrastructure that have the potential to provide sub-millisecond latency.

[0174] Computing at central offices (COs) in the aggregation layer 1450 (sometimes known as the middle edge or far edge layer, depending on distance and latency) can serve as aggregation points for multiple base stations within a local area. For example, one CO can aggregate traffic from approximately 30 base stations. This number can vary by country and population density. These central offices can then link (e.g., by wired links, including optical) to regional points of presence (POPs) before connecting to regional exchange sites. COs can also bring wireless and wired services together. The latency to reach a CO satisfies many edge use cases (e.g., power, space, ease of management), making them a desirable location for edge services. A central office or exchange site can also aggregate multiple central office connections.

[0175] 15 illustrates the deployment and mapping of workloads to operational layers (e.g., corresponding to layers 1420-1470) of edge computing system 1500. Within this configuration, as various collaborations are performed to bring computational resources to workload data and workload data to computational resources, multiple considerations and capabilities are evaluated regarding the location and type of workload execution among the devices of edge computing system 1500. Considerations can include:

[0176] (1) Selecting the appropriate platform architecture, rack design, or other hardware features or configurations for short-term and long-term use (in addition to performing appropriate mapping of services and workloads) according to each location's constraints (e.g., power, space, platform security). Different options may be mapped to different architectural configurations.

[0177] (2) Determine what requirements emerge from the network or service operator to shape the architecture, which may indicate a platform architecture that meets the operator's requirements (e.g., capital versus operational costs, form factor, security, and QoS).

[0178] (3) Determine the right software architecture to manage, monitor, and orchestrate the edge computing architecture. Without the right interfaces to orchestrate, cloud architectures in complex distributed scenarios will not work. Furthermore, exposing the right abstractions and interfaces for services to access the underlying hardware resources is equally important.

[0179] Based on these and other determinations, various workload and use case models 1510 may first be mapped between locations of edge computing systems 1500 according to workload mapping definitions 1520. Such workload mapping definitions 1520 may identify platform requirements and architecture elements 1530 (e.g., storage or rack-scale design techniques 1531, acceleration 1532, platform 1533, fabric or network 1534, storage or memory 1535) that can be deployed between edge computing systems 1500. Additionally, workload mapping definitions 1520 may be based on requirements mappings 1540 that address aspects such as security 1541, physical constraints 1542, management 1543, cost (e.g., monetary, resource, or other asset costs) 1544, infrastructure limitations, and capacity 1545.

[0180] Additionally, end-to-end considerations of the edge computing system 1500 may include evaluation of real-time orchestration, service level agreements (SLAs), and QoS characteristics as defined in definition 1550 (e.g., definition of customer and workload requirements 1551, definition of multi-tier orchestration requirements 1552, definition of base station, central office, and data center locations 1553, 1554, 1555), or definition of E2E QoS or SLA configuration 1556. These definitions 1550 may be used to select platform requirements and architecture elements 1530, rank or prioritize requirements mapping 1540, and ultimately modify workload mapping 1520. These considerations, described in definition 1550 or elsewhere, may reflect the following characteristics:

[0181] 1) Latency. This is used to provide the first key performance indicator (KPI) that plays a key role in edge service location. Since the speed of light is approximately 300,000 km / s and transmission over wires is ~2 / 3 of that, the required response latency will determine how far a device can be from the edge. For example, if some services require a response latency of less than 4 ms, they cannot be more than 150 km from the device. Therefore, for some workloads (e.g., IoT device data processing), a specific edge definition may be used only by base stations, while others may be used by central offices.

[0182] 2) Data privacy, sovereignty, and confidentiality. These are used to determine compliance and validate operability. These considerations may dictate that some services can only exist in specific locations at the edge. For example, in the healthcare segment, some hospitals may want to host and share some services on the edge cloud but without data that crosses certain boundaries of their infrastructure (e.g., on-site equipment, central office, etc.).

[0183] 3) Backhaul Traffic Reduction. Savings on backhaul traffic data can be achieved by filtering traffic at different edges of the network to reduce OPEX / TCO (equivalently, less bandwidth CAPEX may be required on the backhaul network). In this case, filtering can occur at any of the different potential edges of the infrastructure. For example, video surveillance can be processed at the base station to identify which images should be sent to the cloud or the central office, where a content delivery network may be located in the central office.

[0184] 4) Enabling new edge processing use cases, such as services at the edge that accept biometric authentication, or services where payments are made in real time using voice analysis, as long as reliability requirements are met.

[0185] 5) The definition and use of resource-level trust, which allows for authentication of access to capabilities across platforms and resources.

[0186] Determining where the actual edge for a particular use case or workload resides is directly related to the KPIs or value proposition that a particular location offers. For example, defining edge computing for IoT or AR / VR workloads in the core of an operator's infrastructure may not be possible to meet KPI requirements in terms of latency. Therefore, edge computing for this workload would be located closer to the device (at a base station or a more local central office, near or middle layer). On the other hand, edge computing for content delivery network (CDN) (also known as a "content distribution network" or "content-defined network") workloads could be located at a base station, central office, or some other intermediate aggregation point (POA or POP) in the operator's infrastructure (at a middle or far-end layer). In this case, determining what the most appropriate edge location is, along with the associated OPEX / TCO, can derive what the best location for a CDN workload is.

[0187] In a further example, advanced forms of workload mapping may be used in edge computing systems to map specific types of computing activity to specific locations and systems (or to map to types of systems and location capabilities, to more efficiently bring workload data to available computing resources). Figure 16 shows the mapping of workload types to service features for an edge computing system having access points or small cells 1620, gateways or base stations 1640, and a central office 1660 (each with its own capabilities).

[0188] Using computing in the small cell 1620, a combination of network functions 1612 and services 1614 may be provided, with an emphasis on running local or ultra-low latency services (e.g., augmented reality, IoT, FaaS) to generate service results 1672. Using computing in the base station 1640, a similar combination of network functions 1632 and services 1634 may be provided, with the amount of available hardware processing resources in the base station 1640 increasing as the amount and complexity of the network functions increases. Using computing in the central office 1660 (or other aggregation location) may provide deeper layer network functions 1652 that complement services 1654 (e.g., video analytics, storage, analytics, FaaS) that require additional computing resources not available at the access point / small cell 1620 or gateway / base station 1640.

[0189] Some considerations for the location and type of hardware distributed to edge locations 1620, 1640, 1660 and similar subsystems may include:

[0190] (1) Where workloads and use cases are mapped. This decision can be made using various criteria or value propositions discussed herein. Once the mapping is done, different use cases or workloads require decomposition into foundational or basic blocks. Basic blocks can be defined by algorithmic logic units (e.g., deep neural networks or fast Fourier transforms). Once the mapping and decomposition of basic blocks is done at different levels of the edge, specific blocks for improvement can be identified at a given location. Therefore, the respective resource requirements can be used to evaluate how many resources are needed at that specific location.

[0191] (2) Characteristics of each location. As mentioned above, each location (e.g., base station) has a list of physical requirements (e.g., form factor, power, temperature, etc.) as well as a list of the number of expected subscribers (e.g., for a base station, this could range from 1-4K subscribers). The physical requirements translate into how many resources can be deployed at a given location, and the subscribers translate into how many operations are required for a particular workload mapping and subscriber count. Therefore, these and other factors can be important when deploying edge computing processing resources at infrastructure locations (e.g., small cells, base stations, COs).

[0192] A relevant design point for these and other edge computing scenarios, especially in multi-tenant and multi-stakeholder use cases, is that networking infrastructure services cannot "starve" or "fail" and must remain unaffected by ongoing applications and services. Network traffic and network function workloads may need to remain deterministic, so the design of edge cloud architectures can focus on high-priority use cases such as VNFs and network services.

[0193] Figure 17 illustrates workload type mapping to execution platforms in an edge computing system. As illustrated, a set of workload types 1710 (represented as types 1700, 1711, 1712, 1713, and 1714) progresses from type classifications indicating the lowest workload priority and latency requirements (e.g., for IoT data processing) to type classifications indicating the highest workload priority and latency requirements (e.g., for network communication functions), along with intermediate type classifications (e.g., AI workloads, video analytics workloads, and AR / VR workloads). These workload types 1710 can be coordinated within a multi-stakeholder, multi-tenant edge system according to their type classifications.

[0194] Each type classification can be associated with a set of requirements 1720 that can specify workload requirements 1721 for a particular classification (e.g., performance requirements, functional requirements) compared to operator requirements or constraints 1722 (number of available platforms, form factor, power, etc.). As a result of the requirements 1720 for a invoked workload, a selection can be made for a particular configuration of execution platform 1730 for the workload. The configuration of the execution platform 1730 for the workload (e.g., configurations 1731, 1733, 1735 provided by hardware 1732, 1734, 1736) can be selected by identifying the execution platform from among multiple edge nodes (e.g., platforms 1-N); by reconfiguring the execution platform within a configurable rack-scale design system; or by reconfiguring the execution platform through pooling or combining resources from one or more platforms.

[0195] In addition to the requirements and constraints provided by the workload type mapping, other metrics or indicators may be used to select or configure an edge execution platform. For example, mapping services to a specific execution platform can take into account: KPI performance benefits or user experience benefits (e.g., what latency is required to provide a good user experience for 360-degree video); OPEX / TCO (e.g., what is derived from providing a service to a specific location and the expected monetization thereof); and SLA and service level objective (SLO) definitions. These considerations balance the operator's interest in managing potentially high management costs (e.g., high monetary or resource costs) across a distributed ecosystem and different hardware locations.

[0196] FIG. 18 illustrates the operation of services for multiple tenants across multiple layers of an edge computing hardware configuration in edge computing system 1800. Various combinations of hardware availability and platform features are exposed in various operational layers of edge computing system 1800 (e.g., corresponding to layers 1420-1470). For example, a small cell operating in local layer 1420 may have limited hardware (e.g., a low-power CPU) with limited or unspecialized platform functionality (software or hardware capabilities). An on-premises cloudlet / edgelet / or other applet machine operating in on-premises layer 1430 may host additional or more powerful hardware and provide software or hardware features (e.g., AI accelerators, FPGAs, GPUs, cryptographic services, etc.). The base station layer 1440 may have more hardware capabilities (e.g., high-performance CPUs or specialized computing architecture processing units) or more advanced platform capabilities (advanced storage and memory), and a more advanced combination of hardware and platform capabilities (including smart networking components) may be provided in the aggregation layer 1450 and core network layer 1460.

[0197] The various types of hardware capabilities and features illustrated in system 1800 can enable multiple edge FaaS variations. Specifically, a particular service or service platform ("Service A") may be virtually provided for use or execution at any of layers 1420-1460, but various combinations of hardware and software between layers enable various processing results or actions. Furthermore, various combinations of hardware and software (or capabilities of such hardware and software) can be provided for service use or execution based on a particular tenant or user. In this context, the service execution / runtime can be an LSM or other security policy enforcement point. (Similarly, in this context, hardware abstraction layers below the service layer and platform capabilities that enable physical partitioning or virtualization can also provide LSM and other security policy enforcement points.)

[0198] From an application perspective, there may be applications specifically designed for edge networking (e.g., where components of the application run in the cloud and individual processing components reside at the edge within the edge cloud along a hierarchical edge). Thus, the approach illustrated in system 1800 may support multiple variations of FaaS, such as ultra-low latency FaaS and its counterparts, as part of the same or different applications.

[0199] Figure 19 illustrates a further mapping of edge computing hardware configurations to operational deployments and latencies at the network layer, based on the mapping of hardware platforms 1902-1908 (and platform types 1901-1909) to various layers 1430-1450 of the edge cloud and beyond (expanding on the operational network layer example described above with respect to Figure 4). For example, at layer 1430, a combination of a low-power CPU and multiple specialized accelerators (hardware 1902) can provide a first platform type suitable for running on-premise services (e.g., cloudlets / edgelets / or other applets requiring extremely low latency, sub-milliseconds). At layer 1440, a similar combination of a low-power CPU and specialized accelerators (hardware 1904) can provide a second platform type suitable for low-power execution of services for multiple types of devices (e.g., those requiring low latency, sub-5 ms). Deeper in the network, a combination of server-class CPUs with specialized GPUs and accelerators (hardware 1906) or storage (hardware 1908) may be provided at the aggregation layer 1450. Finally, beyond the edge cloud, multi-core server CPUs and storage (hardware 1910) may be provided at the core network layer 1460, leveraging server-class (cloud) processing but at the tradeoff of higher latency. It will be appreciated that in some scenarios, the hardware platform or configuration may operate on a modifiable platform, such as to enable the use of swappable or interchangeable hardware components at different layers of the edge cloud. In this way, a service provider or hardware system operator can upgrade their edge system deployment with newer / more powerful hardware (or more cost-effective hardware) while still being able to serve edge workloads with the required network latency.

[0200] Figure 20 shows a further mapping of use cases and workloads to operational deployments of edge computing hardware configurations. Specifically, Figure 20 shows how various workloads associated with edge clouds can be deployed by service providers, each with different requirements, applications, and value propositions.

[0201] Different types of use cases and workloads may be mapped to different platform types based on the selection or reconfiguration of the hardware configuration. For example, a flexible NFV workload 2010 may be mapped to a first platform type offering CPU and storage resources; a video processing or video analytics workload 2020 may be mapped to a second platform type offering low-power CPU and specialized GPU and FPGA processing; an AR / VR and gaming workload 2030 may be mapped to a third platform type offering CPU and storage resources; a data caching and storage gateway workload 2040 may be mapped to a fourth platform type offering low-power CPU and storage resources; Internet of Things processing 2050 may be mapped to a fifth platform type offering low-power CPU and AI acceleration resources; an autonomous vehicle workload 2060 and a Function as a Service workload 2070 may be mapped to sixth and seventh platform types offering CPU, storage, and specialized GPU processing resources; and a speech recognition workload 2080 may be mapped to an Nth platform type having CPU and storage resources, and specialized GPU processing resources.

[0202] Thus, because both computational resources are mapped to workload data and workload data instances are mapped to computational resources, different locations can be used to perform service management across the edge cloud 110. In a highly distributed architecture, the feature is based on mapping services on the base station. In this case, the physical requirements of the platform in terms of power and space largely limit the amount of hardware that can be deployed at this particular edge node. Furthermore, acceleration methods such as hardware inference acceleration may be utilized to achieve greater service density. In a central office architecture, the architecture is less distributed according to the central office's functions and service locations, but it is also less power- and space-constrained. In this case, with fewer space and power constraints, the architectural solution can be more homogenous, at the expense of some performance or service density.

[0203] It should be understood that the initial workload mapping may not be valid for run-time activities during the workload's lifecycle or workflow configuration. An additional service that should be made available is workload assessment-as-a-service, which can provide workload assessment and reallocation based on workload characterization over time. Based on this, the workload can be moved to another location or to another hardware or system configuration to support the workload needs, as suggested in the following example.

[0204] As a further example, various distribution, upgrade, and change architectures can be implemented to support software (and firmware and hardware feature) updates for workloads and edge computing services generally. Typically, the computing platform vendor is responsible for creating feature changes or security patches that are applied to deployed platforms. The vendor typically cannot enable other supply chain entities to develop firmware updates and / or apply them. While this scenario is also applicable to edge computing environments, distributed computing environments can enable new software distribution and upgrade dynamics. When workloads are analyzed and distributed across multiple platforms, i.e., "slices" or "flavors" of resources across multiple administrators and vendors, consideration may need to be given to whether users and orchestrators have sufficient control over which software / firmware versions are deployed.

[0205] In one example, a workload may be validated or simulated in a particular configuration and deployment "flavor," where the results of the simulation may be entirely dependent on firmware, software, and other configuration parameters. In some cases, security vulnerabilities in the hardware, firmware, and software also predict how the workload execution will behave. However, if the environment used to validate and / or simulate the workload execution differs from the actual environment in which it will run, the difference represents additional risk.

[0206] The edge computing ecosystem may be optimized to minimize risk variations by managing software, firmware, and hardware feature updates. A three-phase approach to workload deployment (1)-(3) can be used: (1) Set up a workload validation environment that identifies runtime dependencies. This considers which software models are required to process the workload application. This dependency graph is identified as part of the validation environment setup. Furthermore, excess functionality presents an increased attack surface that adds runtime risk. These dependencies can be removed from the validation environment. (2) Simulation creates the actual environment required to process the workload. This can include the use of simulated hardware, virtualization, or simulated performance scenarios. The workload is run with predicted interactions with other workloads, orchestration, users, collaboration, etc. This simulation ensures operational corner cases are identified. The simulation can also specify which versions of hardware, software, and firmware will be used. These can be actual hardware, software, and firmware resources to better understand expected actual behavior. (3) The simulation environment is replicated in a real-world deployment. Hardware, software, and firmware versions are adjusted appropriately. Perhaps this means moving backward or backwards to discover and allocate resources according to the environment defined in the simulation. This may include removing hardware, software, and firmware not used by the workload.

[0207] C. Hardware Components FIG. 21A illustrates a first edge computing hardware configuration that maps various architectural aspects of edge platform 2110 (e.g., computing hardware 2111 and 2112, network functions 2113, platform acceleration functions 2114, power management functions 2115, memory 2116, storage 2117, etc.) to specific edge platform capabilities 2120 (e.g., I / O pooling 2121, acceleration pooling 2122, memory pooling 2123, acceleration techniques 2124, storage capabilities 2125). To provide the edge cloud configuration as a holistic solution for a service, workloads or basic hardware components relative to the service and service requirements / constraints (e.g., network and I / O, platform acceleration, power) are considered in terms of available architectural aspects (e.g., pooling, storage, etc.).

[0208] Within edge platform capability 2120, specific acceleration types can be configured or identified among multiple functions to ensure service density is met across the edge cloud. Specifically, four primary acceleration types can be deployed in an edge cloud configuration: (1) general acceleration (e.g., FPGA) for implementing basic blocks such as fast Fourier transforms (FFTs), k-nearest neighbor algorithms (KNNs), and machine learning workloads; (2) image, video, and transcoding accelerators; (3) inference accelerators; and (4) encryption and compression-related workloads (e.g., implemented with Intel® QuickAssist® technology). Thus, the specific design or configuration of edge platform capability 2120 (e.g., general acceleration 2141, inference acceleration 2142, storage 2143) can consider the appropriate type of acceleration and platform product model to be selected to accommodate service and throughput density, as well as available power.

[0209] Figure 21B illustrates a second edge computing hardware configuration, providing a second edge platform 2130 (e.g., having hardware 2131, 2132, 2133, 2134, 2135, 2136, and 2137, as suggested above) along with a second set of edge platform functions 2140 (e.g., having functions 2141, 2142, and 2143) that can be deployed for a lower-power, but higher-service-density solution. This approach aims to define a lower-power solution that uses acceleration techniques to achieve better service density or service throughput per watt. This key design tradeoff results in a platform that sacrifices general-purpose compute in favor of specialized hardware (e.g., FPGAs, inference accelerators) that can perform the same work at a better performance-per-watt ratio. In this example, a "service-density" solution could enable more service actions per platform and per watt, or result in more throughput at the service-per-watt level.

[0210] The platform capabilities 2140 can be designed to be advantageous in terms of both power envelope and physical space. As a result, the configuration of FIG. 21B can provide a suitable target for base station deployment (e.g., in the near or middle-edge layer). However, the platform capabilities 2140 may have tradeoffs, including: (1) orchestration, maintenance, and system management requirements (which can translate into OPEX / TCO costs); and (2) requiring the operator ecosystem to ensure that the entire system stack can work with the various accelerators that are revealed. However, these drawbacks may be mitigated by developing software abstraction layers.

[0211] FIG. 21C illustrates a third edge computing hardware configuration, providing a third edge platform 2150 (e.g., having hardware 2151, 2152, 2153, 2154, 2155, 2156, and 2157, as suggested above) and a third set of edge platform capabilities 2160 (e.g., having functions 2161, 2162, and 2163) that provides a high-power, yet uniform and generic architecture. FIG. 21C offers the opposite approach to FIG. 21B, providing a platform architecture with reduced heterogeneity in the various types of resources that an operator or edge owner must handle in terms of management, maintenance, and tuning. However, eliminating accelerators in favor of uniformity comes at the expense of less service density and service throughput per watt at the platform level. In a further example, edge platform capabilities 2160 can implement general-purpose acceleration (e.g., in the form of an FPGA).

[0212] Other variations of the edge platform shown in Figures 21A-21C may also be adapted. For example, the platform can be sized and designed to incorporate new components that make services and throughput denser but remain more homogenous, for example, by including accelerators within the platform or on-die, allowing operators to manage them seamlessly.

[0213] In further examples, any of the computing nodes or devices described with reference to the present edge computing systems and environments may be implemented based on the components shown in Figures 22A and 22B. Each edge computing node may be embodied as a device, appliance, computer, or other type of "thing" capable of communicating with other edge, networking, or endpoint components. For example, an edge computing device may be embodied as a smartphone, mobile computing device, smart appliance, in-vehicle computing system (e.g., a navigation system), or other device or system capable of performing the described functions.

[0214] 22A , edge computing node 2200 includes a computing engine (also referred to herein as “computing circuitry”) 2202, an input / output (I / O) subsystem 2208, data storage 2210, a communications circuitry subsystem 2212, and optionally one or more peripheral devices 2214. In other examples, each computing device may include other or additional components, such as those typically found in a computer (e.g., a display, peripheral devices, etc.). Furthermore, in some examples, one or more of the illustrated components may be incorporated into or otherwise form a part of another component.

[0215] The computational node 2200 may be embodied as any type of engine, device, or collection of devices capable of performing various computational functions. In some examples, the computational node 2200 may be embodied as a single device, such as an integrated circuit, embedded system, field programmable gate array (FPGA), system-on-chip (SOC), or other integrated system or device. In the illustrated example, the computational node 2200 includes or is embodied as a processor 2204 and memory 2206. The processor 2204 may be embodied as any type of processor capable of performing the functions described herein (e.g., executing an application). For example, the processor 2204 may be embodied as one or more multi-core processors, a microcontroller, or other processor or processing / control circuitry. In some examples, the processor 2204 may be embodied as, include, or be coupled to an FPGA, an application-specific integrated circuit (ASIC), reconfigurable hardware or hardware circuitry, or other specialized hardware that facilitates the performance of the functions described herein.

[0216] The main memory 2206 may be embodied as any type of volatile (e.g., dynamic random access memory (DRAM)) or non-volatile memory or data storage capable of performing the functions described herein. Volatile memory may be a storage medium that requires power to maintain the state of data stored by that medium. Non-limiting examples of volatile memory may include various types of random access memory (RAM), such as DRAM or static random access memory (SRAM). One particular type of DRAM that may be used in the memory module is synchronous dynamic random access memory (SDRAM).

[0217] In one example, the memory device is a block-addressable memory device, such as one based on NAND or NOR technology. The memory device may also include a three-dimensional cross-point memory device (e.g., Intel® 3D XPoint® memory) or other byte-addressable, write-in-place, nonvolatile memory device. The memory device may refer to the die itself and / or the packaged memory product. In some examples, 3D cross-point memory (e.g., Intel® 3D XPoint® memory) may have a transistorless, stackable, cross-point architecture in which memory cells are located at the intersections of word lines and bit lines, are individually addressable, and bit storage is based on changes in bulk resistance. In some examples, all or a portion of the main memory 2206 may be integrated into the processor 2204. The main memory 2206 may store various software and data used during operation, such as one or more applications, data operated by the application(s), libraries, and drivers.

[0218] The computational circuitry 2202 is communicatively coupled to other components of the computational node 2200 via an I / O subsystem 2208, which may be embodied as circuits and / or components that facilitate input / output operations with the computational circuitry 2202 (e.g., processor 2204 and / or main memory 2206) and other components of the computational circuitry 2202. For example, the I / O subsystem 2208 may be embodied as or otherwise include a memory controller hub, an input / output control hub, an integrated sensor hub, a firmware device, a communication link (e.g., a point-to-point link, a bus link, a wire, a cable, an optical guide, a printed circuit board trace, etc.), and / or other components and subsystems that support input / output operations. In some examples, the I / O subsystem 2208 may form part of a system-on-chip (SoC) and be integrated into the computational circuitry 2202 along with one or more of the processor 2204, the main memory 2206, and other components of the computational circuitry 2202.

[0219] The one or more illustrated data storage devices 2210 may be embodied as any type of device configured for short-term or long-term storage of data, such as, for example, memory devices and circuitry, memory cards, hard disk drives, solid-state drives, or other data storage devices. Each data storage device 2210 may include a system partition that stores data and firmware code for that data storage device 2210. Each data storage device 2210 may also include, for example, one or more operating system partitions that store data files and executable programs for the operating system, depending on the type of compute node 2200.

[0220] The communications circuitry 2212 may be embodied as any communications circuitry, device, or collection thereof capable of enabling communications over a network between the computing circuitry 2202 and other computing devices (e.g., the edge gateway node 312 of the edge computing system 300). The communications circuitry 2212 may use one or more communications technologies (e.g., wired or wireless communications) and associated protocols (e.g., cellular networking protocols such as the 3GPP 4G or 5G standards, wireless local area network protocols such as IEEE 802.11 / Wi-Fi®, wireless wide area network protocols, Ethernet®, Bluetooth®, Bluetooth Low Energy, IoT protocols such as IEEE 802.15.4 or ZigBee®, low power wide area network (LPWAN), or low power wide area (LPWA) protocols, etc.) to achieve such communications.

[0221] The illustrated communications circuitry 2212 includes a network interface controller (NIC) 2220, which may also be referred to as a host fabric interface (HFI). The NIC 2220 may be embodied as one or more add-in boards, daughter cards, network interface cards, controller chips, chipsets, or other devices that may be used by the computing node 2200 to connect with other computing devices (e.g., the edge gateway node 312). In some examples, the NIC 2220 may be embodied as part of a system-on-chip (SoC) that includes one or more processors, or may be included on a multi-chip package that also includes one or more processors. In some examples, the NIC 2220 may include a local processor (not shown) and / or local memory (not shown), both of which are local to the NIC 2220. In such examples, the local processor of the NIC 2220 may be capable of performing one or more of the functions of the computing circuitry 2202 described herein. Additionally or alternatively, in such examples, the local memory of the NIC 2220 may be integrated into one or more components of the client computing node at the board level, socket level, chip level, and / or other level.

[0222] Additionally, in some examples, each computing node 2200 may include one or more peripheral devices 2214. Such peripheral devices 2214 may include any type of peripheral device found on a computing device or server, such as, for example, audio input devices, displays, other input / output devices, interface devices, and / or other peripheral devices, depending on the specific type of computing node 2200. In further examples, computing node 2200 may be embodied by a respective edge computing node in an edge computing system (e.g., client computing node 302, edge gateway node 312, edge aggregation node 322), or a similar form of equipment, computer, subsystem, circuit, or other component.

[0223] In a more detailed example, FIG. 22B illustrates a block diagram of example components that may be present within an edge computing node 2250 to implement the techniques (e.g., operations, processes, methods, and methods) described herein. This edge computing node 2250 provides a closer view of each component of node 2200 when implemented as or as part of a computing device (e.g., a mobile device, a base station, a server, a gateway, etc.). The edge computing node 2250 may include any combination of the hardware or logical components referenced herein and may include or be coupled to devices usable with an edge communications network or a combination of such networks. These components may be implemented as, or portions thereof, ICs, discrete electronic devices, or other modules, instruction sets, programmable logic or algorithms, hardware, hardware accelerators, software, firmware, or combinations thereof, adapted within the edge computing node 2250, or may be implemented as components otherwise incorporated within a larger system chassis.

[0224] The edge computing device 2250 may include processing circuitry in the form of a processor 2252, which may be a microprocessor, a multi-core processor, a multi-threaded processor, an ultra-low voltage processor, an embedded processor, or other known processing element. The processor 2252 may be part of a system-on-chip (SoC) in which the processor 2252 and other components are formed into a single integrated circuit or a single package (e.g., an Edison® or Galileo® SoC board from Intel Corporation of Santa Clara, California). By way of example, the processor 2252 may include a CPU processor from the Intel® Architecture Core® family, such as a Quark®, Atom®, i3, i5, i7, i9, or MCU class processor, or other such processor available from Intel®. However, any number of other processors may be used, such as those available from Advanced Micro Devices, Inc. (AMD®) of Sunnyvale, California, MIPS®-based designs from MIPS Technologies, Inc. of Sunnyvale, California, and ARM®-based designs licensed from ARM Holdings, Inc. or its customers, or their licensees or adopters. Processors may include units such as the A5-A13 processors from Apple®, the Snapdragon® processors from Qualcomm® Technologies, Inc., and the OMAP® processors from Texas Instruments. Processor 2252 and associated circuitry may be provided in a single-socket form factor, a multiple-socket form factor, or a variety of other formats, including limited hardware configurations or configurations that include fewer than all of the elements shown in FIG. 22.

[0225] The processor 2252 may communicate with system memory 2254 over an interconnect 2256 (e.g., a bus). Any number of memory devices may be used to provide a given amount of system memory. By way of example, the memory may be random access memory (RAM) according to a Joint Electron Devices Engineering Council (JEDEC) design, such as DDR or mobile DDR standards (e.g., LPDDR, LPDDR2, LPDDR3, or LPDDR4). In particular examples, memory components may conform to DRAM standards promulgated by JEDEC, such as JESD79F for DDR SDRAM, JESD79-2F for DDR2 SDRAM, JESD79-3F for DDR3 SDRAM, JESD79-4A for DDR4 SDRAM, JESD209 for low-power DDR (LPDDR), JESD209-2 for LDDR2, JESD209-3 for LDDR3, and JESD209-4 for LDDR4. Such standards (and similar standards) are sometimes referred to as DDR-family standards, and communication interfaces of memory devices implementing such standards are sometimes referred to as DDR-family interfaces. In various implementations, individual memory devices may be in a number of different package types, such as a single-die package (SDP), a dual-die package (DDP), or a quad-die package (QDP). In some instances, these devices may be soldered directly onto the motherboard to provide a lower profile solution, while in other instances, these devices are instead configured as one or more memory modules that are coupled to the motherboard by a given connector. A number of other memory implementations may also be used, such as other types of memory modules, for example, dual in-line memory modules (DIMMs) in various variations, including, but not limited to, micro-DIMMs or mini-DIMMs.

[0226] Storage 2258 may also be coupled to processor 2252 via interconnect 2256 to provide persistent storage of information, such as data, applications, the operating system, etc. In one example, storage 2258 may be implemented by a solid-state disk drive (SSDD). Other devices that may be used for storage 2258 include flash memory cards, such as SD cards, microSD cards, XD picture cards, and the like, as well as USB flash drives. In one example, the memory device may be or include a chalcogenide glass-based memory device, a multi-threshold level NAND flash memory, a NOR flash memory, a single-level or multi-level phase change memory (PCM), a resistive memory, a nanowire memory, a ferroelectric transistor random access memory (FeTRAM), an antiferroelectric memory, a magnetoresistive random access memory (MRAM) incorporating memristor technology, a resistive memory including metal oxide-based, oxygen vacancy-based, and conductive bridge random access memory (CB-RAM), or a spin-transfer torque (STT)-MRAM, a spintronic magnetic junction memory-based device, a magnetic tunnel junction (MTJ)-based device, a DW (domain wall) and SOT (spin orbit transfer)-based device, a thyristor-based memory device, or any combination thereof, or other memory.

[0227] In low-power implementations, storage 2258 may be on-die memory or registers associated with processor 2252. However, in some examples, storage 2258 may be implemented using a micro-hard disk drive (HDD). Also, any number of newer technologies may be used for storage 2258 in addition to or instead of the technologies mentioned above, such as resistive memory, phase change memory, holographic memory, or chemical memory, among others.

[0228] These components may communicate over interconnect 2256. Interconnect 2256 may include any number of technologies, including Industry Standard Architecture (ISA), Extended ISA (EISA), Peripheral Component Interconnect (PCI), Peripheral Component Interconnect Extended (PCIx), PCI Express (PCIe), or any number of other technologies. Interconnect 2256 may be, for example, a proprietary bus used in SoC-based systems. Other bus systems may also be included, such as an I2C interface, an SPI interface, a point-to-point interface, and a power bus, among others.

[0229] The interconnect 2256 may couple the processor 2252 to a transceiver 2266 for communication with a connected edge device 2262. The transceiver 2266 may use any number of frequencies and protocols, such as 2.4 gigahertz (GHz) transmissions under the IEEE 802.15.4 standard, using the Bluetooth® Low Energy (BLE) standard or the ZigBee® standard defined by the Bluetooth® Special Interest Group, among others. Any number of wireless devices configured for a particular wireless communication protocol may be used to connect to a connected edge device 2262. For example, a wireless local area network (WLAN) unit may be used to implement Wi-Fi® communications according to the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard. Additionally, wireless wide area communications, for example according to cellular or other wireless wide area protocols, may occur via a wireless wide area network (WWAN) unit.

[0230] The wireless network transceiver 2266 (or multiple transceivers) may communicate using multiple standards or radios for communication at different ranges. For example, the edge computing node 2250 may communicate with nearby devices, e.g., within about 10 meters, using a local transceiver based on BLE or other low-power radio to conserve power. Connected edge devices 2262 that are more distant, e.g., within about 50 meters, may be reached with ZigBee or other medium-power radios. Both communication technologies may occur on a single radio at different power levels, or may occur on separate transceivers, e.g., a local transceiver using BLE and a separate mesh transceiver using ZigBee.

[0231] A wireless network transceiver 2266 (e.g., a radio transceiver) may be included for communicating with devices or services in the edge cloud 2290 via local or wide area network protocols. The wireless network transceiver 2266 may be an LPWA transceiver conforming to the IEEE 802.15.4 or IEEE 802.15.4g standards, among others. The edge computing node 2250 may communicate over a wide area using LoraWAN® (Long Range Wide Area Network) developed by the Semtech and LoRa alliance. The techniques described herein are not limited to these technologies and may be used with any number of other cloud transceivers implementing long-range, low-bandwidth communications, such as Sigfox, and other technologies. Additionally, other communication techniques may be used, such as time-slotted channel hopping as described in the IEEE 802.15.4e specification.

[0232] In addition to the systems mentioned for wireless network transceiver 2266 as described herein, any number of other wireless communications and protocols may be used. For example, transceiver 2266 may include a cellular transceiver using spread spectrum (SPA / SAS) communications to implement high-speed communications. Additionally, any number of other protocols may be used, such as Wi-Fi networks for medium-speed communications and for providing network communications. Transceiver 2266 may include radios compatible with any number of 3GPP (Third Generation Partnership Project) specifications, such as Long Term Evolution (LTE) and 5th generation communications systems, as described in further detail at the end of this disclosure. A network interface controller (NIC) 2268 may be included to provide wired communications to nodes of edge cloud 2290 or other devices, such as connected edge devices 2262 (operating within a mesh). The wired communication may provide an Ethernet connection or may be based on other types of networks, such as, for example, a Controller Area Network (CAN), a Local Interconnect Network (LIN), DeviceNet, ControlNet, Data Highway+, PROFIBUS, or PROFINET, among others. A further NIC 2268 may be included to allow connection to a second network, for example, a first NIC 2268 providing communication to the cloud over Ethernet and a second NIC 2268 providing communication to other devices over another type of network.

[0233] Given the various types of applicable communications from the device to another component or network, applicable communications circuitry used by the device can include or be embodied by any one or more of components 2264, 2266, 2268, or 2270. Thus, in various examples, applicable means for communicating (e.g., receiving, transmitting, etc.) can be embodied by such communications circuitry.

[0234] The edge computing node 2250 may include or be coupled to acceleration circuitry 2264, which may be embodied by one or more AI accelerators, neural computer sticks, neuromorphic hardware, FPGAs, arrangements of GPUs, one or more SoCs, one or more CPUs, one or more digital signal processors, dedicated ASICs, or other forms of specialized processors or circuits designed to accomplish one or more specialized tasks. These tasks may include AI processing (including machine learning, training, inference, and classification processing), visual data processing, network data processing, object detection, rule analysis, or the like.

[0235] The interconnect 2256 may couple the processor 2252 to a sensor hub or external interface 2270 used to connect additional devices or subsystems. The devices may include sensors 2272, such as accelerometers, level sensors, flow sensors, optical light sensors, camera sensors, temperature sensors, global navigation system (e.g., GPS) sensors, pressure sensors, barometric pressure sensors, and the like. The hub or interface 2270 may further be used to connect the edge computing node 2250 to actuators 2274, such as power switches, valve actuators, audible sound generators, visual warning devices, and the like.

[0236] In some example options, various input / output (I / O) devices may be present in or connected to the edge computing node 2250. For example, a display or other output device 2284 may be included to show information such as sensor readings or actuator positions. An input device 2286, such as a touchscreen or keypad, may be included to accept input. The output device 2284 may include any number of forms of audio or visual display, including simple visual output such as binary status indicators (e.g., LEDs) and multi-character visual output, or more complex output such as a display screen (e.g., LCD screen), with output such as text, graphics, multimedia objects, and the like generated or generated from the operation of the edge computing node 2250. In the context of the present system, the display or console hardware may be used to provide edge computing system output and receive input, manage edge computing system components or services, identify the status of edge computing components or services, and perform any number of other management or governance functions or service use cases.

[0237] A battery 2276 may power the edge computing node 2250, although in instances where the edge computing node 2250 is mounted in a fixed location, the edge computing node 2250 may have a power source coupled to an electrical grid, or the battery may be used for backup or temporary capacity. The battery 2276 may be a lithium-ion battery or a metal-air battery, such as a zinc-air battery, an aluminum-air battery, a lithium-air battery, and the like.

[0238] If a battery 2276 is included, a battery monitor / charger 2278 may be included in the edge computing node 2250 to track the state of charge (SoCh) of the battery 2276. The battery monitor / charger 2278 may also be used to monitor other parameters of the battery 2276, such as the state of health (SoH) and state of function (SoF) of the battery 2276, to provide failure prediction. The battery monitor / charger 2278 may include a battery monitoring integrated circuit, such as the LTC4020 or LTC2990 from Linear Technologies, Inc., the ADT7488A from ON Semiconductor of Phoenix, Arizona, or an IC from the UCD90xxx family from Texas Instruments of Dallas, Texas. The battery monitor / charger 2278 may communicate information about the battery 2276 to the processor 2252 over the interconnect 2256. The battery monitor / charger 2278 may also include an analog-to-digital converter (ADC) that allows the processor 2252 to directly monitor the voltage of or current flow from the battery 2276. The battery parameters may be used to determine actions that the edge computing node 2250 may take, such as, for example, transmission frequency, mesh network operation, sensing frequency, and the like.

[0239] A power block 2280, or other power source coupled to the power grid, may be coupled to the battery monitor / charger 2278 to charge the battery 2276. In some examples, the power block 2280 may be replaced with a wireless power receiver to obtain power wirelessly, for example via a loop antenna in the edge computing node 2250. A wireless battery charging circuit, such as the LTC4020 chip from Linear Technologies, Inc. of Milpitas, California, among others, may be included in the battery monitor / charger 2278. The particular charging circuit may be selected based on the size of the battery 2276 and therefore the current required. Charging may be performed using the Airfuel standard promulgated by the Airfuel Alliance, the Qi wireless charging standard promulgated by the Wireless Power Consortium, or the Rezence charging standard promulgated by the Alliance for Wireless Power, among others.

[0240] Storage 2258 may include instructions 2282 in the form of software, firmware, or hardware commands for implementing the techniques described herein. Although such instructions 2282 are shown as code blocks contained in memory 2254 and storage 2258, it will be understood that any of these code blocks may be replaced by hardwired circuitry, for example, as incorporated in an application specific integrated circuit (ASIC).

[0241] In one example, instructions 2282 provided via memory 2254, storage 2258, or processor 2252 may be embodied as non-transitory machine-readable medium 2260 containing code that instructs processor 2252 to perform electronic processing at edge computing node 2250. Processor 2252 may access non-transitory machine-readable medium 2260 over interconnect 2256. For example, non-transitory machine-readable medium 2260 may be embodied by the devices described with respect to storage 2258 or may include a specific storage unit such as an optical disk, a flash drive, or any number of other hardware devices. Non-transitory machine-readable medium 2260 may include instructions that instruct processor 2252 to perform a particular sequence of actions or flow, for example, as described with respect to the flowchart(s) and block diagram(s) of the operations and functions described above. As used herein, the terms “machine-readable medium” and “computer-readable medium” are interchangeable.

[0242] In a further example, machine-readable media also includes tangible media capable of storing, encoding, or carrying instructions for execution by a machine, causing the machine to perform any one or more of the methods of the present disclosure, or of storing, encoding, or carrying data structures utilized by or associated with such instructions. Accordingly, "machine-readable media" may include, but is not limited to, solid-state memory, as well as optical and magnetic media. Specific examples of machine-readable media include, but are not limited to, semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices), magnetic disks, such as internal hard disks and removable disks, magneto-optical disks, and non-volatile memory, including CD-ROM and DVD-ROM disks. Instructions embodied by machine-readable media may also be transmitted or received over a communications network using a transmission medium via a network interface device employing any one of a number of transfer protocols (e.g., HTTP).

[0243] A machine-readable medium may be provided by a storage device or other device capable of hosting data in a non-transitory format. In one example, information stored or otherwise provided on a machine-readable medium may represent instructions, such as the instructions themselves or a format from which instructions can be derived. This format from which instructions can be derived may include source code, encoded instructions (e.g., in a compressed or encrypted form), packaged instructions (e.g., divided into multiple packages), or the like. Information representing instructions in a machine-readable medium may be processed by a processing circuit into instructions for implementing any of the operations described herein. For example, deriving instructions from information (e.g., processing by a processing circuit) may include compiling (e.g., from source code, object code, etc.), interpreting, loading, organizing (e.g., dynamically or statically linking), encoding, decoding, encrypting, decrypting, packaging, unpackaging, or otherwise manipulating the information into instructions.

[0244] In one example, deriving instructions may include assembling, compiling, or interpreting (e.g., by processing circuitry) information to generate the instructions from some intermediate or preprocessed format provided by a machine-readable medium. If the information is provided in multiple parts, it may be combined, decompressed, and modified to create the instructions. For example, the information may be multiple compressed source code packages (or object code, or binary executable code, etc.) on one or several remote servers. The source code packages may be encrypted during transport over a network, and then decrypted, decompressed, assembled (e.g., linked), and compiled or interpreted (e.g., into a library, a standalone executable program, etc.) as needed at the local machine and executed by the local machine.

[0245] FIG. 22C is a block diagram of communication components within an example mobile device 2232. This mobile device 2232 provides a more detailed view of the communication processing components of node 2200 or device 2250 when implemented as user equipment or a component of user equipment. Mobile device 2232 may include a radio front-end module (FEM) circuit 2234, a radio IC circuit 2236, and a baseband processing circuit 2238. While the illustrated mobile device 2232 includes both wireless local area network (WLAN) and Bluetooth® functionality, aspects of the device are not so limited, and other wireless technologies described herein may be implemented with similar circuitry. FEM circuit 2234 may include, for example, a wireless LAN or Wi-Fi FEM circuit 2234A and a Bluetooth (BT) FEM circuit 2234B. The WLAN FEM circuit 2234A may include a receive signal path having circuitry configured to operate on WLAN RF signals received from one or more antennas 2231A, amplify the received signals, and provide the amplified versions of the received signals to the WLAN radio IC circuit 2236A for further processing. The BT FEM circuit 2234B may include a receive signal path that may include circuitry configured to operate on BT RF signals received from one or more antennas 2231B, amplify the received signals, and provide the amplified versions of the received signals to the BT radio IC circuit 2236B for further processing. The FEM circuit 2234A may also include a transmit signal path that may include circuitry configured to amplify the WLAN signals provided by the radio IC circuit 2236A for wireless transmission by one or more of the antennas 2231A. In addition, the FEM circuit 2234B may also include a transmit signal path that may include circuitry configured to amplify the BT signals provided by the radio IC circuit 2236B for wireless transmission by one or more antennas 2231B.While FEM2234A and FEM2234B are shown as separate from one another in the example of FIG. 22C, aspects of the disclosure are not so limited and include within their scope the use of an FEM (not shown) that includes transmit and / or receive paths for both WLAN and BT signals, or the use of one or more FEM circuits where at least some of these FEM circuits share transmit and / or receive signal paths for both WLAN and BT signals.

[0246] The illustrated radio IC circuitry 2236 may include a WLAN radio IC circuitry 2236A and a BT radio IC circuitry 2236B. The WLAN radio IC circuitry 2236A may include a receive signal path that may include circuitry for downconverting WLAN RF signals received from the FEM circuitry 2234A and providing baseband signals to the WLAN baseband processing circuitry 2238A. In turn, the BT radio IC circuitry 2236B may include a receive signal path that may include circuitry for downconverting BT RF signals received from the FEM circuitry 2234B and providing baseband signals to the BT baseband processing circuitry 2238B. The WLAN radio IC circuitry 2236A may also include a transmit signal path that may include circuitry for upconverting WLAN baseband signals provided by the WLAN baseband processing circuitry 2238A and providing WLAN RF output signals to the FEM circuitry 2234A for subsequent wireless transmission by one or more antennas 2231A. BT radio IC circuit 2236B may also include a transmit signal path that may include circuitry that upconverts BT baseband signals provided by BT baseband processing circuit 2238B and provides a BT RF output signal to FEM circuit 2234B for subsequent wireless transmission by one or more antennas 2231 B. While radio IC circuits 2236A and 2236B are shown as separate from one another in the example of FIG. 22C, aspects of the disclosure are not so limited and include within their scope the use of radio IC circuits (not shown) that include transmit and / or receive signal paths for both WLAN and BT signals, or the use of one or more radio IC circuits where at least some of these radio IC circuits share transmit and / or receive signal paths for both WLAN and BT signals.

[0247] The baseband processing circuit 2238 may include a WLAN baseband processing circuit 2238A and a BT baseband processing circuit 2238B. The WLAN baseband processing circuit 2238A may include memory, such as a set of RAM arrays, in a fast Fourier transform or inverse fast Fourier transform block (not shown) of the WLAN baseband processing circuit 2238A. The WLAN baseband circuit 2238A and the BT baseband circuit 2238B may each further include one or more processors and control logic for processing signals received from a corresponding WLAN or BT receive signal path of the radio IC circuit 2236 and for generating a corresponding WLAN or BT baseband signal for a transmit signal path of the radio IC circuit 2236. Each of the baseband processing circuits 2238A and 2238B may further include physical layer (PHY) and media access control layer (MAC) circuits and may interface with an application processor 2251 (or, in other examples, processor circuit 2250) for generating and processing baseband signals and for controlling the operation of the radio IC circuit 2236.

[0248] 22C , according to the illustrated embodiment, WLAN-BT coexistence circuit 2243 may include logic that provides an interface between WLAN baseband circuit 2238A and BT baseband circuit 2238B to enable use cases requiring coexistence of WLAN and BT. Additionally, a switch 2233 may be provided between WLAN FEM circuit 2234A and BT FEM circuit 2234B to enable switching between WLAN and BT radios according to application needs. Also, while antennas 2231A, 2231B are shown connected to WLAN FEM circuit 2234A and BT FEM circuit 2234B, respectively, embodiments of the present disclosure include sharing one or more antennas between the WLAN FEM and BT FEM, or providing two or more antennas connected to each of FEM 2234A or 2234B.

[0249] In some aspects of the present disclosure, the front-end module circuitry 2234, the radio IC circuitry 2236, and the baseband processing circuitry 2238 may be provided on a single radio card. In other aspects, one or more antennas 2231A, 2231B, the FEM circuitry 2234, and the radio IC circuitry 2236 may be provided on a single radio card. In some other aspects of the present disclosure, the radio IC circuitry 2236 and the baseband processing circuitry 2238 may be provided on a single chip or integrated circuit (IC).

[0250] In still further examples, the computing capabilities of a computing node may be implemented with computational storage, or "compute-in storage," which refers to a data storage solution implemented with computational capabilities. In various examples, compute-in storage may be implemented as compute offloading in a block storage device, using compute offloading in an object-based storage device, using compute offloading in a distributed storage system, or using compute offloading provided across multiple non-aggregated storage environments. An example compute-in storage system may be provided from one or more storage devices, each having non-volatile memory and a compute offload controller. In this system, the non-volatile memory stores data, and the compute offload controller performs computational tasks on the data based on compute offload commands from a host processor. Among other examples, acceleration capabilities and functions may comprise the use of such compute-in storage configurations.

[0251] Figure 22D illustrates rack-scale design (RSD) components that may be included as part of a server or other discrete compute node in an edge platform architecture. This configuration provides a more detailed view of the configurable processing components of node 2200 or device 2250 when implemented as a server (e.g., in a server rack, blade, etc.). This configurable architecture differs from others by not aggregating field-programmable gate array (FPGA), non-volatile memory express (NVMe), and input-output (I / O) pooling resources. The FPGA and NVMe resources provide elements that can be used for any type of edge service, such as video analytics and audio analytics. I / O pooling can be used to provide flexible network functions. This architecture allows for scaling of network interfaces according to the expected data rate or network load for a particular virtual network function (VNF). This architecture also allows for flexibility in mapping various network cards to compute nodes depending on the type of network processing occurring at a given node.

[0252] The illustrated RSD architecture includes a point-of-delivery (POD) manager 2242. The POD manager 2242 is responsible for managing resources (including compute and non-aggregate resources) within a POD (e.g., one or more racks). The POD manager 2242 exposes an interface to the orchestrator to create, manage, or destroy configured nodes. Managing configured nodes includes the ability to increase or decrease the amount of pooled resources 2248 connected to a particular compute thread 2240. The POD manager 2242 typically runs on a node controller. The POD manager 2242 is responsible for discovering resources within a POD, configuring and managing resources, and configuring logical servers. In one example, the POD manager 2242 is an optional separate component and is not required within a rack. However, in one example, to be "RSD-compliant," a rack can be managed by a certified POD manager.

[0253] The following are some example attributes of POD manager 2242. For example, a rack may include a set of compute threads 2240 used to run edge services and other related system software stacks (e.g., orchestration or other system services). One type of compute thread 2240 may be a pooled resources thread. This compute thread 2240 may manage a set of non-aggregated resources. Here, the compute thread 2240 may include pooled system management engine software (PSME) 2241. The PSME 2241 provides a management interface for managing modules or blades at the drawer level. In one example, a rack houses one or more logical PSMEs. For example, each drawer may have a PSME, or multiple server drawers may share a PSME, or a PSME may run on a top-of-rack (TOR) switch 2244 or on a separate host. In one example, the PSME 2241 supports the RSD API.

[0254] In one example, compute thread 2240 may include multiple processors (e.g., CLX) running an RSD software stack implementing NVM-oF or FPGA-oF that act as a target system and manage a set of non-aggregated resources. In one example, the processors are connected using a PCIe x16 bifurcated port to a PCIe switch 2246 that provides access to the target resources (FPGA or NVME within RSD 2248).

[0255] To run edge services, various RSD edge configuration node flavors can be used by compute threads 2240. Services running on these nodes can use client software libraries or drivers to provide transparent access to the non-aggregated FPGAs and NVMEs in RSD 2248. In a further example, a rack includes one or more PCIE switches that connect compute threads 2240 to a set of non-aggregated resources (e.g., RSD 2248).

[0256] The block diagrams in Figures 22A, 22B, 22C, and 22D are intended to illustrate high-level views of various devices, subsystems, or configuration components of an edge computing node. However, it is understood that other implementations may omit some of the illustrated components, may include additional components, or may result in different configurations of the illustrated components. Furthermore, these configurations may be used in a variety of use cases and environments, including those described below (e.g., mobile UEs in industrial computing for smart cities or smart factories, among other examples).

[0257] Each computing platform in Figures 21A-21C and Figures 22A-22D can support multiple edge instances (e.g., edge clusters) through the use of tenant containers running on a single computing platform. Similarly, multiple edge nodes can exist as multiple subnodes running on multiple tenants within the same computing platform. Thus, based on available resource divisions, a single system or computing platform can be partitioned or divided into supporting multiple tenant and edge node instances, each of which can support multiple services and functions, potentially operated or controlled within multiple computing platform instances by multiple owners. These various types of partitions can support complex multi-tenancy and many combinations of multi-stakeholders through the use of LSMs or other implementations of isolation / security policies. Reference is made to the use of LSMs and security features to enforce or realize such security functions in the following sections. Similarly, services and functions running on these various types of multi-entity partitions can be load balanced, migrated, and coordinated to achieve the required service objectives and processing.

[0258] D. Use cases for edge computing systems Edge computing meets several value propositions and key performance indicators (KPIs) for many types of use cases and deployments involving multi-system computing operations (multi-tenant, multi-user, multi-stakeholder, multi-device, etc.) These value propositions enable improved response latency, increased security, and lower backhaul traffic, all of which enable new use cases.

[0259] Defining where the actual Compute Edge "resides" for a particular use case or workload is directly related to the KPIs or value proposition that a particular location provides. For example, defining the Compute Edge for IoT or Augmented Reality / Virtual Reality (AR / VR) workloads within the core of the operator's infrastructure would not meet the KPI requirements from a latency perspective. Therefore, the Compute Edge for this workload would be located closer to the device (bringing the computational resources closer to the workload data, e.g., in the near-edge or middle-edge tier at a base station or central office). On the other hand, for CDN workloads, the Compute Edge may be hosted at a base station, central office, or other intermediate aggregation point (POA or POP) in the operator's infrastructure.

[0260] The devices (or mesh of devices) used by end users may be considered far-end devices, but many also provide some computational edge capability. Devices in this scenario may provide the lowest possible latency and may function in a mesh or connected IoT network type. However, at some point, the end devices may become computationally limited or may not be as power-efficient as needed to perform a given task. For example, at some point in the network traffic load, the AR / VR user will experience severe degradation (even to the point of performing the workload at the far end of the device itself).

[0261] The top priorities for edge computing between various network layers include central office functions, video-on-demand / content data network services, media analytics, live streaming / transcoding, and AV / VR, including online gaming. Central office workloads may include, for example, SDN / NFV for converged footprints and new service models. Video-on-demand / content data network workloads may include, for example, streaming, surveillance, alarm systems, building access, and the like. Live streaming / transcoding may include enhanced user experience (UX), such as live streaming from social networking applications. AR / VR workloads may include, for example, real-time retail, gaming networks, and the like. Media analytics (including both audio and video) may also be widely deployed within edge deployments. These workloads constitute another important pillar used in many of the aforementioned workloads (e.g., surveillance, AR, etc.).

[0262] In addition to these preferred ones, other use cases may be enabled within the edge cloud. In-memory databases (IMDBs) are increasingly making sense in the design of edge computing deployments. Their target uses are to serve online transaction processing (OLTP, for handling small transaction processing requests, e.g., small reads or writes) and online analytical processing (OLAP, for handling larger analytical processing requests, e.g., larger dataset queries and computational tasks). These two types of workloads mostly map to network analytics to improve infrastructure management or other types of analytics or data storage (e.g., IoT).

[0263] Enterprise and government use cases can also be enabled within the edge cloud, where some enterprise customers are looking at possible edge deployments as a possible way to improve some of their processes. An example of this is thin clients, which can be moved across the edge infrastructure to provide low-latency access to thin clients as employees of a given enterprise move geographically.

[0264] Use cases related to Industry 4.0 initiatives may also be enabled, which may be improved by virtualization in manufacturing, such as in use cases related to ultra-reliable, low-latency communications over 5G. Such communications allow TSP providers to offer more complete solutions for both industrial workloads and localized 5G networks.

[0265] Healthcare is another area where, as technology advances and security techniques mature, hospitals and other healthcare institutions will use edge clouds to securely store and process healthcare-related data. In this setting, edge clouds will facilitate resource sharing and help reduce CAPEX and OPEX / TCO costs.

[0266] Another use case area that can be enabled via the edge cloud includes V2V, V2X, and other types of advanced driver assistance systems (ADAS) workloads. As automotive technology becomes more and more mature, the edge cloud will enable device-to-device communication as well as a means for users at the edge (V2X) to access edge services.

[0267] The following sections describe many different examples of workload use cases related to edge cloud, along with their requirements, use cases, and value propositions to third-party service providers. These sections are broadly categorized into the following areas: data caching, video / video analytics, NFV, RAN, augmented reality / virtual reality (AR / VR), vehicle driver assistance, Internet of Things (IoT), industrial applications, gaming, accelerated browsing, and speech analytics. However, it is understood that some edge cloud use cases or architectures may be relevant under multiple or other categories.

[0268] Use Case - Data Caching

[0269] Data caching refers to caching data at the edge for faster loading on user devices. An example of data caching includes caching popular videos in an area for faster access. Data caching uses storage and computational resources and is typically not latency-bound. Data caching is used in content delivery networks (CDNs). Global mobile data traffic is steadily increasing, and video data now accounts for a large portion of mobile network data traffic. With the distribution of video content, redundant content is likely to be fetched and delivered to users in the same region.

[0270] Based on this scenario, edge clouds become a suitable infrastructure for caching content at the edge of the network, which can significantly reduce backhaul traffic. This has the potential to reduce OPEX / TCO costs for third-party service providers (and even CSPs and TSPs). Content caching can extend beyond video to other areas, such as music and documents.

[0271] Several approaches can be used to perform content caching within the edge cloud:

[0272] (a) Content caching based on traffic learning Cache content in specific regions based on its popularity and increasing requests / traffic, as well as cache content similar to certain popular content as a proactive strategy.

[0273] (b) Targeted Content Caching Targeted caching means caching content at the edge for a targeted audience, for example, for viewers in a stadium or at a rally.

[0274] (c) User-guided caching Users indicate content that should be cached (for a service fee or as part of the user's data plan). A simple example would be a video that a user adds to "watch later" on a video site, or a video that they put in their favorites list on a streaming video service, which may be candidates for caching. Since there may be several users in the same area with similar content interests, caching this content paves the way for saving backhaul traffic costs (monetary and resource costs).

[0275] (d) Service Experience Cashing This functionality is based on the transactional and experience agreement between the user and the provider. For example, if a user wants to obtain 4K video and 9.1 sound data, but the remote and local services cannot handle the throughput, latency, and transcoding requirements, the system should negotiate to pre-fetch the content at the expected QoS, although this may result in delays or suboptimal performance (e.g., delays at the start of playback).

[0276] Content caching as a use case has different workloads that are executed together to perform the caching. Some of these use cases may involve the execution of content caching algorithms, data aggregation procedures, machine learning code, content traffic analysis, web servers, etc.

[0277] This use case is not latency bound, and the end user experience is expected to be positive, as caching increases the chances of servicing a request sooner than would normally be the case.

[0278] This use case may employ a diverse set of workloads that work together to enable caching. Therefore, resource usage may also vary. Content caching algorithms are traditionally optimized for CPUs. However, machine learning or inference code that assists with analysis may use FPGAs. The most utilized resource is storage resources, which are natural users of pooled storage.

[0279] The content caching implementation can be provided in many configurations (e.g., one master process and multiple worker processes). For example, a master process can be used to read and evaluate the configuration and maintain the worker processes as they perform the actual processing of requests. This can be implemented by a content service provider employing an asynchronous and event-based model and OS-dependent mechanisms to efficiently distribute requests among multiple worker processes. The number of worker processes can be specified in a configuration file and be fixed for a given configuration, or it can automatically adjust to the number of available CPU cores. This means that a content service provider can handle multiple requests from one worker process as shown, rather than dedicating a separate process for each connection.

[0280] A content service provider may support multiple protocols (e.g., over HTTP, such as HDS (HTTP Dynamic Streaming by Adobe), HLS (HTTP Live Streaming by Apple), and MPEG-Dash (open source and used by YouTube and Netflix)). Generally, these protocols work by cutting the video stream into chunks and creating metadata and a playlist. The client first downloads the metadata along with the playlist to know which chunks make up the entire video, and then makes HTTP requests to retrieve each chunk. The difference between video on demand (VOD) and live streaming is that in VOD, the chunks and playlists are generated in advance, while in live streaming, the playlist is generated as new chunks are created (after they are read from the stream).

[0281] The throughput of a vCDN can be expressed as the number of parallel streams that the CDN can support with a given hardware and network configuration, for a smooth video viewing experience (e.g., so that the user does not see pauses for content to load while the video is playing).

[0282] In another example, edge cloud CDNs enable cable, telco, and mobile network service providers to build private CDNs to deliver content they or their partners manage to their subscribers with a broadcast TV-like experience, regardless of the type of consumer device or the subscriber's location. Edge cloud CDNs are optimized to support streaming delivery of TV services, including video-on-demand (VoD), live events, and cloud digital video recorders (CDVRs). This can be provided by content service providers who establish two key aspects of a CDN: (a) deploying CSP software distribution nodes (e.g., using commodity hardware at the edge) and (b) connecting the nodes to CSPs. This can be done using an edge cloud configured to enable secure access to the service provider's cloud-based dashboard for managing configuration, operation, and performance.

[0283] In one example, an individual CSP edge node may support three content delivery modes: (a) open caching: delivery of third-party and partner content in both HTTP and HTTPS through delegation and using specifications (e.g., specifications recognized by the Streaming Video Alliance); (b) transparent caching: delivery of partner, over-the-top, service provider-owned HTTP content, for example, through transparent caching (e.g., when transparent caching operates autonomously to discover, categorize, cache, and deliver popular content at the network edge and optimize streaming content across the network); and (c) open edge CDN: delivery of service provider-owned content in both HTTP and HTTPS through edge nodes deployed within the service provider network and managed by cloud-based operations.

[0284] Use Cases - Video / Video Analytics

[0285] Video analytics refers to performing live video analysis and video pre-processing or transcoding at the edge for presentation to user devices. Traffic video analysis and alerting systems are examples of video analytics at the edge. This type of use relies on storage and computational resources.

[0286] Video analytics plays an important role in many fields. For example, facial recognition from traffic or security cameras already plays a key role in law and order. Several other types of analytics can also be performed on video content, such as object tracking, motion detection, event detection, fire and smoke detection, and AI learning of patterns in live streams or video archives. Currently, video analytics is performed on the cloud or on dedicated private servers depending on the needs and capabilities to be established. Performing video analytics at the edge brings both demands and opportunities for several areas, such as: For example:

[0287] Surveillance and Public Safety: For example, near-instant processing of live video streams at the edge can lead to better surveillance and law and order enforcement. Face detection and incident identification and trigger generation are some of the functions that can be performed at the edge. These functions can enable law enforcement officials to take emergency action regarding an incident.

[0288] Supporting connected cars and autonomous driving: For example, a live stream of a scene viewed by an autonomous vehicle needs to be analyzed within a very short time frame to determine the action the car should take. Autonomous driving may already include the resources to process the scene instantaneously. Edge video analytics can serve to process (or pre-process) more distant scenes or post-process footage for continuous training and feedback.

[0289] Enabling Smart Cities and IoT: Video analytics at the edge is a key element for enabling smart cities. For example, traffic video analytics can be used to route traffic along the most efficient paths. The detection of fire or smoke in an area can be instantly identified and feedback sent to both the city infrastructure and connected cars in the specific area to ensure traffic does not continue towards the danger zone.

[0290] Enhanced infotainment services: Video analytics at the edge can be used to enhance the real-time audience experience at events such as sports, concerts, and other shows. Footage from multiple camera angles at an event can be analyzed, AR / VR features applied, and presented to live audiences via large screens, smartphones, VR devices, etc.

[0291] The latency and turnaround time requirements for video analytics at the edge vary for different scenarios, as illustrated in Table 1 below. [Table 1]

[0292] TSPs have several ways to monetize video analytics services delivered at the edge: a. Improved surveillance and monitoring can be offered as a service to government agencies and private security companies. b. Video analytics for smart cities (and its specific capabilities) can be offered as a service to both government agencies, IoT device providers, and households, as needed. c. Video analytics for enhanced infotainment can be offered as a service to event spectators or organizers (serving on behalf of spectators).

[0293] Use Cases - Network Functions Virtualization (NFV) and Radio Access Network (RAN) Operations

[0294] Flexible NFV relies heavily on storage and compute resources (e.g., CPU or FPGA). Edge cloud deployments can control the user experience by co-hosting network (NFV) and service functions on the same platform / co-location. Being able to place local breakouts closer to the servers that actually provide the requested service can provide better response times and reduce OPEX / TCO by reducing unnecessary traffic through the infrastructure or data center.

[0295] Regarding the integration of MEC and network function virtualization (NFV) within the same network deployment, as per the ETSI specification deployment, MEC and NFV are complementary concepts that can exist independently. The MEC architecture is designed to allow for multiple different deployment options for MEC systems. With regard to NFV, the MEC system can be realized independently of or coexist with the presence of an NFV environment within the same network. Because both MEC and NFV may utilize virtualization technology, MEC applications and NFV virtualized network functions can be instantiated partially or fully on the same virtualization infrastructure.

[0296] The MEC reference architecture in an NFV environment reuses the concepts of a virtualized infrastructure manager, similar to the Virtualized Infrastructure Manager in the NFV framework (with some augmentations), and a virtualized infrastructure, which roughly corresponds to the NFV Infrastructure Point-of-Presence (NFVI-PoP) described in the NFV specification. The reference architecture is designed so that further synergies between MEC and NFV can be achieved.

[0297] Depending on the operator's network preferences and migration strategy, multiple deployment scenarios are possible, such as fully virtualized or mixed environments, MEC-first or NFV-first deployments, different levels of integration between the two technologies, and consideration of secondary aspects such as multi-tenancy. The way in which the MEC and NFV management and orchestration components interact (e.g., integration, interoperation, coexistence) is a key aspect of integrated MEC-NFV deployments. RAN use cases also include the use of decentralized RANs (e.g., at the edge) and UPFs. As is well understood, the RAN relies heavily on storage and computational resources (e.g., CPUs or FPGAs). Further discussion of RAN architectures is provided below in the 5G connectivity example.

[0298] Use Cases - Augmented Reality / Virtual Reality (AR / VR)

[0299] Most Augmented Reality (AR) use cases involve image recognition (faces, objects, etc.) and analytics on them. AR (and VR) usage is expected to focus on user-controlled endpoints, including between endpoints interacting with each other, as supported by the edge. These anticipated usages often require low latency to fulfill service requests, whether involving a single endpoint device, a group of devices, or a device-to-device approach.

[0300] The client device involved in AR is typically in the form of a camera-equipped mobile phone or wearable. A typical usage scenario is for a user to point the camera device at an object and view useful annotations about that object. Several smartphone apps already exist that provide AR services, such as ViewAR, Vivino, and Augment, and such apps or associated wearables are designed to provide AR for specific purposes. For example, Vivino provides useful information about a wine bottle when a photo of the label is captured and provided. Similarly, ViewAR assists with 3D visualization for packaging and space planning of a room.

[0301] An AR scenario typically consists of three stages: (i) transmitting captured images from the device, (ii) image recognition and object identification, and (iii) performing analysis to generate results (in the form of annotations with useful information found from the Internet, object identification, etc.). Of the above stages, (ii) and (iii) are generally the most time-consuming. A specialized AR service specifies specific targets to the image recognition service, which identifies images and searches for those targets. The AR service then uses the results to perform further processing, such as analysis or finding more information from the Internet. These results are then displayed to the user.

[0302] AR services typically use a separate service for image recognition. For example, many of the apps mentioned above use Vuforia as their main engine for image recognition. The AR service pre-specifies targets to the Vuforia engine through its API, allowing image recognition to find the specified targets. Image recognition can be performed both on the device and in the cloud. Performing it on the device using an on-device database is not power-efficient, while performing the same in the cloud takes longer. Another example is the Catchoom cloud image recognition service, which allows AR apps to send requests to their service in the cloud via a RESTful API, which responds with image recognition. After the image is recognized, the AR app performs a third process from the client device, which is to search for the desired information on the Internet and present it to the user.

[0303] Image recognition services such as Vuforia and Catchoom have several AR apps that use their services. This is expected to increase in the future only as more such AR apps are created and the number of users of these AR apps increases. Because these services are currently hosted in the cloud, the latency and turnaround time for image recognition is significant. For example, finding information about wine in Vivino takes several seconds. This becomes unacceptable in scenarios involving wearables, requiring time-critical annotation, and where real-time movements need to be taken into account. Therefore, moving image recognition services from the cloud to the edge can improve the overall turnaround time and provide users with a seamless experience.

[0304] As mentioned above, current solutions require several seconds to start generating the first annotation or first output to the user. This delay is referred to as "response time." This is primarily due to the fact that they are cloud-hosted services. For a seamless user experience and to support future time-critical AR use cases, the turnaround time requirements are very short. Table 2 below summarizes these requirements according to various sources. [Table 2]

[0305] Image recognition, object tracking, and identification are the most computationally intensive parts of an AR service. The most well-known image recognition algorithms are SURF, SIFT, FAST, and BRIEF. Services such as Vuforia and Catchoom do not disclose the core image recognition algorithms they use or their resource requirements. However, any image recognition algorithm can be implemented using both CPUs and FPGAs. The volume of requests these services encounter can determine the size of the CPU or FPGA available. For example, Vuforia has registered over 300 apps, and Catchoom has already performed 750 million interactions to date, with a projected number of interactions reaching 10 billion by 2025. Therefore, as the volume of requests increases, the system requires more parallelism to fulfill incoming requests and therefore more resources. These services also use storage to store image repositories for object matching. This can also be located at the edge, allowing image recognition and object identification to be performed directly from the edge without an internet connection. In an advanced scenario, the AR service app may also use storage at the edge to store cached data from the internet to provide a faster response to the user as soon as an image is recognized. The cached data may also be located in different tiers, such as in-memory, persistent memory on NVRAM, or pooled SSD storage.

[0306] Third-party service providers (TSPs) can monetize AR services at the edge. The edge can host a central image recognition and object identification service that can be used by AR apps. The AR app specifies a target through an API, and the service can respond with the desired object when an input request is sent. The app will pay for the service at the edge.

[0307] The edge can host image recognition services such as Vuforia or Catchoom, where you pay for the resources used, and have a separate business model from the apps that use those services - for example, Catchoom SaaS can be delivered from the edge.

[0308] Use Cases - Vehicle Assisted and Autonomous Driving Operations

[0309] Edge computing nodes can be used to assist autonomous vehicle operations. Example use cases include, but are not limited to, navigation, vehicle-to-vehicle (V2V) communications, vehicle-to-infrastructure (V2I) communications, or vehicle-to-everything (V2X) communications. Latency is a factor in vehicle assistance use cases to ensure near real-time response times for autonomous driving.

[0310] ETSI MEC is developing V2X MEC services to support V2X interoperability in multi-vendor, multi-network and multi-access environments. It describes V2X related information flows, required information and operations. The MEC specification specifies the required APIs along with data models and data formats. ETSI MEC is a standardized solution for C-V2X edge cloud deployments, therefore, with this in mind, MEC030 WI introduces a new MEC service, later called "V2X Information Service" (VIS).

[0311] Automotive use cases involve multiple stakeholders, including various car manufacturers, OEM (original equipment manufacturer) suppliers, network infrastructure vendors, MEC vendors, application / content providers, and other stakeholders.

[0312] More specifically, according to the requirements in ETSI MEC, multi-access, multi-network and multi-operator scenarios are the reference assumptions that motivate the need for MEC normative work in this field, including scenarios applicable to V2X services. For example, some V2X services may be managed by the OEM (so-called "vehicle OEM scenarios"), and therefore it makes sense to consider both single-operator and multi-operator scenarios for such services. It is expected that V2X services will be provided by several different network operators in the same country and / or in several different countries.

[0313] The same is true when considering an "ITS operator scenario" where the operator may provide services to multiple different vehicle OEMs. An Intelligent Transportation System (ITS) operator may need to provide a nationwide V2X service by leveraging different operator networks (deploying different MEC systems) and providing this service to vehicles belonging to different OEMs. Again, it is expected that V2X services will be provided by multiple network operators in the same country and / or in multiple different countries.

[0314] As a result, to enable all use cases, VIS services running in the edge cloud can support C-V2X systems implemented in common scenarios, especially those scenarios that presuppose the existence of multiple MEC vendors and the need to enable interoperable data exchange between them.

[0315] Furthermore, multi-operator interoperability is a consideration to ensure service continuity for V2X. More specifically, in a typical multi-operator scenario, there can be numerous scenarios involving temporary lack of radio coverage (e.g., in roaming situations). Also, in traditional V2X systems, interconnection between mobile network operators (MNOs) is remotely terminated, which entails obvious drawbacks in terms of high end-to-end (E2E) latency. On the other hand, thanks to the utilization of VIS services (which also enable "horizontal communication" between MEC systems), interconnection between MNOs can be achieved with low E2E latency.

[0316] V2X service continuity must be ensured across all territories including the coverage areas of both operators, and there must be no service interruption and guaranteed E2E performance when leaving one operator's coverage area and entering the other. To this end, the VIS manages the multi-operator environment by publishing information about PC5 configuration parameters, especially when the UE is out of coverage.

[0317] MEC deployment depends on the definition of "edge." The MEC standard allows several options to provide MNOs with the necessary flexibility, especially when deploying MEC within an NFV environment (in practice, MEC entities would be instantiated as virtualized network functions (VNFs) in this case, thus offering operators greater flexibility in terms of deployment). Therefore, in principle, the edge cloud is exposed everywhere, allowing for flexible deployment of MEC depending on the use case, vertical segment, or information being processed. Furthermore, some components of an MEC system are better aligned when co-located with other elements of the system. For example, in certain use cases (e.g., enterprise), MEC apps may need to consume MEC services locally, and therefore it may be worthwhile to deploy an MEC host with the required set of APIs locally. In other cases, deploying an MEC server in a data center (far from the access network) may eliminate the need to host some APIs, such as the RNI API (which collects radio network information from radio base stations). On the other hand, RNI information is accounted for and made available to a Cloud RAN (CRAN) environment at an aggregation point, thus enabling the execution of suitable radio-aware traffic management procedures. In some other cases, it may make sense for bandwidth management APIs to exist both at the access network level and at more remote locations, for example, to properly set up transport networks for Content Delivery Network (CDN)-based services.

[0318] Use Cases - Internet of Things (IoT)

[0319] Internet of Things (IoT) devices are found in many sectors, such as factory automation, process automation, smart grid, smart retail, consumer devices, automotive, home automation, office and commercial applications, security, military, and other applications. The IoT places great demands on communication response times, computational resources, and storage resources. Latency requirements depend on the application. For example, a factory automation environment may require round-trip latency of 0.25 ms to 10 ms, a smart grid implementation may use round-trip latency of 3-20 ms, and a process automation environment may use 50-100 ms. The network topology used in an IoT network can include any of the technologies described herein, in addition to a mesh network provided using a Bluetooth® Low Energy (BLE) link, a wireless local area network (WLAN) used to communicate with IoT devices via an IEEE 802.11 (Wi-Fi®) link, a cellular network used to communicate with IoT devices via an LTE / LTE-A (4G) or 5G cellular network, or a low-power wide-area (LPWA) network, such as an LPWA network compatible with the LoRaWan specification promulgated by the LoRa Alliance or an IPv6 over a low-power wide-area network (LPWAN) network compatible with specifications promulgated by the Internet Engineering Task Force (IETF). Additionally, each IoT network may communicate with external network providers (e.g., Tier 2 or Tier 3 providers) using any number of communication links, such as an LTE cellular link, an LPWA link, or a link based on the IEEE 802.15.4 standard (e.g., Zigbee®). Each IoT network may also operate using a variety of network and internet application protocols, such as the Constrained Application Protocol (CoAP).Each IoT network may also be integrated with a coordinator device that provides a chain of links that form a cluster tree of linked devices and networks.

[0320] The Internet of Things (IoT) is expected to be both a significant producer and consumer of data. IoT network nodes can include hardware Root of Trust (e.g., using the DICE architecture) that attests to node configurations to peer nodes, which in turn attest to aggregate subnetworks and topologies to next-tier proof verifiers. This structure roughly matches the nesting and topology dynamics of edge servers, FaaS, meshes, and virtual tenant "slices," and thus, it can be understood, to use common, distributed, generalized proof verification and analysis capabilities to assess security at the top level, edge-to-edge, or subdivided into regional, local, or other partitioning schemes.

[0321] IoT components typically include elements / devices from smart cities, smart retail, smart vehicles, smart homes, etc. In a sense, the workloads mentioned above, such as video analytics and AR / VR, are also part of IoT, since edge devices are theoretically IoT devices. For example, a face detection workload may be performed for devices in a smart city environment, or for checkout in a smart retail store, or as part of AR for individual users. Thus, IoT workloads generally include all AI workloads in that they process specific data points.

[0322] A more specific IoT-related workload is the IoT gateway. IoT devices are expected to generate trillions of gigabytes of data every day in the near future. All this data needs to be processed differently with different latencies for different purposes. This also means that computational power is needed to process all this data in different locations (to meet different latency requirements). The edge cloud is an ideal place to perform (a) data pre-processing (bidirectional), e.g., filtering and reformatting, (b) data processing for latency-critical use cases and scenarios with connected components, and (c) partial data processing and storage.

[0323] Data organization and processing are also operations in edge clouds. Essentially, data organization entities vary in complexity, from simple key-value stores designed for very fast access to data to complex analytical operations. The following are broad classifications of data organization and processing software that may be enabled in edge clouds: (a) NoSQL key-value stores / caches: Store data for very fast lookup and retrieval, but do not perform any processing on the data (e.g., Aerospike, Redis). (b) NoSQL column stores: Store data for very fast lookup and retrieval, and perform some basic operations, such as range lookups (e.g., HBase, Cassandra). (c) NoSQL document stores: Perform operations such as search and index-based lookups (e.g., MongoDB, Marklogic). (d) NoSQL graph databases: Perform specialized operations related to graph traversal. Example: Neo4j. (e) NewSQL / SQL databases: Perform a vast range of complex database queries. (f) Analytics software: Perform statistical analysis on data from various stores.

[0324] Some workloads and enterprise workloads that have traditionally been deployed in the cloud will eventually be deployed at edge locations as well. These include workloads from the following categories:

[0325] Low-latency or contextualized data distribution: In this case, some existing workloads deployed in the cloud can be deployed at edge locations to provide faster access to data in a more secure manner (without having to traverse multiple security domains) and with less jitter in terms of data access (without having to traverse multiple layers, from the telco infrastructure to end servers, perhaps hosted at the edge). A first example of a workload is an in-memory database, such as HANA or Spark-SQL, and a data front end, such as a web server. This use case can be combined in some cases with accelerated web browsing. A second example of a workload is data mining or analytics at the edge, where existing data mining workloads can be used at edge locations to perform real-time or post-mortem data analysis. These use cases can involve real-time analytics to derive dynamic or adaptive solutions, which can be used, for example, for adaptive network functions or dynamic IoT management schemes. The main value proposition of this use case from an edge perspective is the latency of real-time decision-making. These use cases can also involve near real-time or post-event data processing, which can be used, for example, to process all of the data that IoT devices generate to the edge, which does not necessarily need to be stored and processed on the network backhaul. The main value proposition of this use case from an edge perspective is backhaul traffic savings and computation distribution.

[0326] This edge cloud projection strategy can aid in platform definition by projecting edge cloud sizing for different platforms, offered services, and number of users. This can have the effect of projecting edge cloud sizes (number of nodes) implied by different services offered at the edge, projecting edge cloud sizing with and without accelerators, e.g., FPGAs, for different services, and projecting power requirements for different platforms. A projection model can be utilized for this purpose. For example, a projection model can perform the projection by analyzing workload characteristics representing each service / use case. For each workload, a brief description of the workload, modeling assumptions, and data and configurations used by the model can be considered by the projection model.

[0327] Many of the AI ​​models used in the evaluation are part of the OpenVINO toolkit, a toolkit for developing and deploying CNN-based deep learning inference. The toolkit provides common software tools and optimization libraries that support heterogeneous execution of AI models across CPUs, GPUs, FPGAs, and computer vision accelerators such as Movidius Neural Compute Sticks. The OpenVINO toolkit also includes a set of pre-trained AI models optimized to run on the underlying hardware, some of which are used in the evaluation and analysis of workloads. Using this toolkit, for example, video analysis can be performed using the appropriate AI model or a combination of models depending on the scenario.

[0328] Use Case - Discovery

[0329] IoT: Face Detection ADAS - This is a standard face detection model that is used for a variety of purposes, such as monitoring whether passengers are inside a car and counting pedestrian traffic indoors. It also finds use in driver assistance warning systems. For example, it can be used to continuously scan the driver for signs of fatigue and tiredness and alert the driver with visual and audible warnings to bring their attention back to the road when necessary.

[0330] IoT: Face Detection - This model is typically used in retail environments, for example, to identify known shoplifters, identify people committing return fraud, etc. Furthermore, it can be combined or followed by models that support retail analytics, such as: (a) Age and Gender Recognition (using a model that detects the age and gender of a detected face. The detected face from the face detection model is provided as input to this model, which estimates the person's age and gender. In the retail domain, it can be used for analytics, for example, to understand what categories of people look at and purchase what items. This model can also be used in other domains, such as video surveillance); and (b) Head Pose Estimation (using a model that detects the position of a person's head). In a retail environment, this can be used to understand what caught a person's attention. This can also be used in ADAS or video surveillance.

[0331] IoT: People Detection - People detection is used to detect people in both indoor and outdoor environments using cameras installed at higher vantage points. This can be used for a variety of purposes, e.g., crowd counting, video surveillance, etc.

[0332] IoT: People, Vehicles, and Bicycle Detection - This model is used to analyze video from outdoor cameras, for example to distinguish between people, bicyclists, bicyclists only, and vehicles. The model's variable lighting conditions improve accuracy in daylight, darkness, and varying weather conditions.

[0333] Use Case - Healthcare Industry Applications

[0334] Industrial applications can be in the healthcare field, manufacturing, retail, or other industries. In healthcare, edge computing can be used to implement various medical devices through connectivity and analytics. One example use case is telesurgery, where doctors can operate on patients remotely. Such use cases involve high computational, communication, and storage resources.

[0335] In another context, with more deaths and disabilities attributable to noncommunicable diseases (e.g., NCDs, cancer, and diabetes), it is important to devise regular screening methods for broad populations. Given the dual problem of fewer professionals and more people residing in rural areas than in urban areas in many countries, it is important to seek solutions that address these issues. Artificial intelligence (AI)-based technologies have shown great success in terms of the quality of NCD / CD screening, and many countries around the world have begun adopting this technology to supplement professionals and thereby address large-scale deployment (e.g., the UK) and to provide access to these services through AI-assisted remote screening in rural areas (e.g., China). AI-assisted (remote) screening can significantly enhance early intervention for many NCDs (and CDs), which would otherwise deplete a country's backbone: human resources. Early screening for NCDs / CDs helps provide better treatment plans and a better quality of life after treatment. These services can then be made available in rural and remote areas without requiring patients to travel long distances for screening. These AI-assisted disease screenings can be based on imaging techniques (X-ray, ultrasound, fundus, OCT, PET, MRI, infrared, CT) and non-imaging techniques (e.g., physiological body parameters such as height, weight, BMI, heart rate, ECG, blood pressure, blood sugar, etc.).

[0336] Another aspect that is slowly but steadily gaining attention is the need to move from disease care to health care. This requires continuous, real-time analysis of potentially high-risk individuals who can be monitored for early warning signals. This can be achieved through wearable medical devices that can continuously monitor specific physiological parameters to alert healthcare providers with timely input. Another use case is the continuous, real-time analysis of elderly individuals who can be monitored for early warning signals. These analyses also include audiovisual data in addition to physiological parameters monitored through wearable devices. A final use case centers on continuous, real-time analysis of physiological parameters captured from sick patients recovering in home or hospital care. Recent FDA approval of certain AI-based disease diagnostic solutions opens the door to the application of AI technology in the highly protected medical field.

[0337] Edge-deployable healthcare analytics can be categorized into three segments regarding the application of AI technology: (a) disease screening through AI; (b) multimodal disease diagnosis through AI for accurate diagnostic quality; and (c) precision medicine through AI for improving intervention efficacy through targeted therapy.

[0338] There are several aspects to the workflow in the healthcare analytics field: besides the scope to update AI models based on new evidence or evidence, there is a constant effort to ensure that the quality of screening / diagnosis is of utmost importance, i.e. false negatives should be 0 and false positives tend to be 0. This means that an AI engine capable of inference can also have the capability to perform incremental model updating (retraining) based on new evidence or evidence.

[0339] Inference latency is a very important parameter, as the performance of the AI ​​engine is much faster than that of the expert, so the expert can get timely input from the AI ​​engine for final diagnosis.

[0340] Since many diagnostic modalities are image-based, there is a need for better image compression without the loss of critical disease biomarkers. Also, healthcare data is strictly controlled in terms of usage. This requires that there be sufficient support for healthcare data privacy / protection.

[0341] AI techniques, such as object detection and segmentation, offer unique possibilities for helping radiologists identify problems more quickly and accurately, which can result in better case prioritization, better outcomes for more patients, and reduced financial costs for hospitals. However, AI for medical imaging is often challenging because the information is often high-resolution and multidimensional. Due to memory constraints, downsampling images to lower resolutions can result in misdiagnoses unless biomarkers are preserved. Once an AI model is trained to an acceptable level of accuracy, it needs to be incorporated into the imaging architecture. Given how large radiology images typically are, it is crucial to be able to process these images efficiently without slowing down the radiologist's workflow or affecting the model's accuracy.

[0342] For example, a bone age prediction model takes input from an X-ray image of a human bone, such as a wrist, along with the patient's gender. An inference model then determines an estimated age from the bone to help identify medical conditions that lead to osteoporosis. For example, if a young patient's estimated age is lower than their chronological age, the patient may be suffering from malnutrition.

[0343] Another example is a lung segmentation model that identifies the lungs in a patient's chest CT scan and then creates a segmentation mask around the detected organ. The results can be used to measure lung size and volume or to load organ-specific disease screening models, such as for tuberculosis or pneumothorax detection. Furthermore, isolating the lungs in an image allows radiologists to have a clearer anatomical depiction of the organ without being distracted by other organs.

[0344] Use Case - Accelerated Browsing

[0345] Accelerated browsing refers to web page-related pre-processing at the edge to prepare and send pages or other content to a user device. Examples of accelerated browsing include web page rendering, ad blocking, content rating, and the like. Accelerated browsing is processor resource intensive (e.g., uses CPU and FPGA resources).

[0346] Edge-accelerated web is a use case that enables edge cloud services to be used for all smartphone users. In typical networks, page load times are dominated by front-end processing rather than servers. Browsers perform processing such as content evaluation and rendering, which consumes not only time but also power, which is crucial for power-critical end devices such as mobile phones. By performing these processing operations at the edge, users can experience a refined browsing experience while conserving their device's battery power. Processing can include ad blocking, rendering, content evaluation, video transcoding, etc.

[0347] Although this use case is not latency-bound, various requests may cause the edge server to delay processing and thereby slow down the load time of the web page. In particular, the edge server may be adapted to handle a large number of HTTP requests and deliver a defined QoS for each request.

[0348] Use Case: Speech Analytics

[0349] As Verbal User Interfaces (LUI) become established as a more natural way of interfacing with users, more speech analysis applications will touch us all. Chatbots are one example. Below we provide an example of speech analysis in the (edge) cloud (server usage).

[0350] In one example, two people are having a conversation in two different languages. This conversational speech usage involves several speech analysis components: voice (or speech) recognition, which converts speech to text in one language; machine translation, which converts text in one language to text in a second language; and speech synthesis, which converts text to speech in the second language. Associated requirements include high quality, real-time execution, and low latency.

[0351] Speech analysis can be used in a wide variety of user interfaces across several use case areas. For example, there are several application areas for speech analysis, including but not limited to digital assistants, virtual call centers, healthcare and (social) media, automotive, and security. Within different application areas, there are several possible uses.

[0352] As another example, the digital assistant field includes voice messaging, voice search, voice dialing, voice memos, voice command, voice navigate, and voicemail.

[0353] As another example, the virtual call center sector includes the telecommunications sector, the financial sector, the retail sector, the health sector, the transportation sector, the e-commerce sector, the hospitality sector, and other sectors where speech analysis may be used.

[0354] Another example in the healthcare field includes medical transcription, medical assistants (e.g., chatbots), medical analytics (e.g., text, charting, etc.), and electronic health records with voice recognition.

[0355] As another example, the (social) media application field includes voice chat, automatic subtitling (e.g. as done by YOUTUBE®), content discovery (search based on voice input), language translation, insight into media traffic to trigger appropriate advertising (as a possible business means), and media summarization.

[0356] Another example is voice command and voice navigation in the automotive field.

[0357] Another example in the security field includes voice recognition (eg, biometric security), voice passwords, and voice control.

[0358] Although there are several uses for each of the application areas described herein and several application areas for speech analysis, the core components of speech analysis can be used across several areas: 1) automatic speech recognition (ASR), 2) machine translation (MT), 3) text-to-speech (TTS), and 4) natural language understanding (NLU) or natural language processing (NLP).

[0359] Some of the commonly occurring uses of speech analysis consist of primitives from speech analysis: ASR, MT, TTS, NLP, and Biometrics. These uses can vary in terms of requirements. Typical requirements for these uses include throughput (the number of simultaneous streams that can be handled by given hardware), latency (in seconds), and quality. The uses can have widely varying requirements regarding latency, such as conversational speech analysis cannot tolerate end-to-end latency greater than ½ second, while text transcription can have delays on the order of several seconds (2-4 s). All use cases have requirements for the highest quality (for acceptance of speech analysis as an AI aid) and the highest throughput (to provide a cost-effective solution).

[0360] Besides throughput, latency, and quality, there may be requirements related to model size (leading to vocabulary size), which is expected to range from a few MB to hundreds of MB (based on current technology, but the size is likely to increase), supported languages / dialects / accents, data privacy / security needs, and mixed-mode analysis, where integrated analysis of image / video / audio / speech / text data is performed. Additionally, edge clouds cater to special-purpose needs such as low vocabulary size and low latency, and therefore may benefit from acceleration, and there are requirements regarding flexibility versus efficiency that may drive the use of accelerators over CPUs.

[0361] To provide AI-enabled speech analysis in contact centers, it is important for solution providers to deploy their AI-based solutions on a flexible platform that offers high performance and scalability without increasing financial or resource costs. An end-to-end deployment of a virtual voice assistant brings together diverse solution components, including deep neural network algorithms, traditional speech processing, graph processing, enterprise software integration, and more. An efficient platform architecture for such deployments can provide leading performance across these workload types and accommodate real-time latency rather than throughput. This platform architecture can be powered by a scalable CPU (e.g., Intel® Xeon® Scalable CPU) used for AI inference deployment, so that the performance and scalability benefits are delivered to the AI-enabled speech analysis workload without increasing the complexity of the deployment environment.

[0362] Speech analytics includes technologies such as continuous speech recognition (CSR), natural language processing (NLP), text-to-speech (text-to-speech), and voice biometrics. Traditionally, speech analytics has been applied offline. For example, a manager might record customer interactions, select a few conversations, examine them, and spot-check to infer how well an agent adheres to company guidelines. AI-based speech analytics enables businesses to respond in real time. Voice recognition, teamed with speech analytics, allows organizations to monitor many more conversations, even up to 100%. Looking ahead, these support services could pave the way for bot assistants that offer customers self-service with an unconstrained natural language interface before handing the call over to a trained agent.

[0363] Although speech analytics use cases closely follow the requirements of conversational interfaces found in consumer devices, such as smart speakers or mobile-based voice assistants, some unique differences exist. Speech analytics solutions for contact centers tend to be very domain-specific and not very general-purpose. Most speech analytics vendors build in flexibility to customize their solutions based on the industry vertical. Their solutions also need to support multiple regional languages. Therefore, the diversity of algorithms deployed can be considerable.

[0364] III. Improved Resource Allocation and Deallocation for Multi-Stakeholder Edge Computing Systems In a further example, the advanced resource allocation and deallocation methods may be extended to diverse multi-stakeholder edge computing environments.

[0365] Edge Heartbeat Example

[0366] One consideration in edge deployments is service assurance. Edge computing is unique in that there can be tens or hundreds of thousands of network (or edge) devices (e.g., drones, sensors, user equipment (UE), or other computing devices) that communicate information with edge services (e.g., computing services running on a service coordinating entity). Having a large number of network devices means that failures are bound to occur in edge deployments. For example, sensors may fail at a certain rate, and such device failures need to be taken into account to evaluate the viability of edge services, but edge services collect data from multiple sources, including network devices.

[0367] The following heartbeat tracking techniques can be used in conjunction with heartbeat, beacon, and tracking capabilities to manage device connectivity for multiple edge nodes or edge devices (e.g., nodes 302, 312, 322) within an edge computing network (e.g., the edge cloud 110 and edge computing system configurations shown in FIGS. 3 through 22D ) in connection with facilitating service assurance. Typically, a “heartbeat” applies to tracking when a device, service, or application is down or unresponsive due to latency issues. A location “beacon” is used when a device, tower, server, etc. broadcasts its location so that a receiver can determine its location via triangulation or otherwise. Location “tracking” applies to devices that share location information (or obtain it from a third party) so that peers can create a log of visited locations. The following description of heartbeat services integrates these types of reporting aspects into a more robust version of heartbeats applicable to edge computing environments.

[0368] More specifically, a service coordinating entity (e.g., a multi-access edge computing (MEC) host, such as one implemented in accordance with the MEC architectures depicted and described with respect to FIGS. 7 through 12) executing a computing service (e.g., the edge services and functions depicted and described with respect to FIGS. 7 through 12) may include an edge heartbeat component configured to perform one or more of the following functions: monitoring and ensuring that a sufficient number of edge devices are active and communicating with the computing service; determining where edge devices are located and tracking them (e.g., whether an edge device is turned on or off, whether an edge device is active or inactive with respect to communication with the computing service, whether an edge device is associated with the computing service based on device location, latency, geo-fencing, etc.); and providing notifications in connection with managing service guarantees for the computing service (e.g., providing notifications when the number of edge nodes communicatively coupled to or actively communicating with the computing service is below or above a threshold number). These notifications may be provided in the form of messages and message exchanges, stored data values, interface operations, and the like.

[0369] FIG. 23 illustrates a low-connectivity cluster of edge nodes and connectivity adjustments based on notifications by an edge heartbeat component, according to some aspects. Referring to FIG. 23, a diagram 2300 illustrates multiple edge nodes in a low-connectivity configuration 2302. More specifically, the edge nodes (shown as dots in FIG. 23) may be coupled to each other and to edge services running on a service coordinating entity (e.g., an MEC host, not shown in FIG. 23) via existing (or active) connections 2306. FIG. 23 further illustrates possible (or inactive) connections 2308 between the edge nodes.

[0370] In some aspects, an edge heartbeat component (e.g., as shown in connection with FIG. 25) is configured to monitor the number of connections between edge nodes and edge services, and between edge nodes, and generate a notification when the number of connections falls below a connectivity threshold.

[0371] 23 illustrates an example of a connectivity threshold of K=2, which indicates the maximum number of active connections per edge node. Low connectivity configuration 2302 is associated with some edge nodes, e.g., 2310, 2312, and 2314, having a single or no active connection between themselves or with an edge service. After an edge heartbeat component notification of low connectivity for an edge service, connectivity is adjusted and the edge nodes transition to normal connectivity configuration 2304, where each network node, e.g., 2316, 2318, and 2320, has at least one or two active connections to the edge service or to another node. However, the generalized attestation function may provide heartbeat-like activity and other feedback when heartbeat results have high accuracy, security, clarity, etc.

[0372] FIG. 24 illustrates a high-connectivity cluster of edge nodes and connectivity adjustments based on notifications by an edge heartbeat component, according to some aspects. Referring to FIG. 24, a diagram 2400 illustrates multiple edge nodes in a high-connectivity configuration 2402. More specifically, edge nodes (shown as dots in FIG. 23), such as 2310-2320, may be coupled to each other and to edge services running on service coordinating entities (e.g., MEC hosts, not shown in FIG. 24) via existing (or active) connections 2406. FIG. 24 further illustrates possible (or inactive) connections 2408 between edge nodes.

[0373] In some aspects, an edge heartbeat component (e.g., as shown in connection with FIG. 25 ) is configured to monitor the number of connections (e.g., network operations, logical operations) between edge nodes and edge services, and the number of connections between edge nodes, and generate a notification when the number of connections exceeds a connectivity threshold.

[0374] FIG. 24 illustrates an example connectivity threshold of K=2, which indicates the maximum number of active connections per edge node. A high connectivity configuration 2402 is associated with some edge nodes having more than one active connection among themselves or with an edge service (not shown in FIG. 24). After an edge heartbeat component notification of high connectivity for an edge service, connectivity is adjusted and the edge nodes transition to a normal connectivity configuration 2404, where each network node has at least one active connection or two active connections to the edge service or to another node. In yet a further example, such heartbeats may be deployed within a cluster (e.g., a logical or physical group or domain) of edge nodes.

[0375] FIG. 25 illustrates an example edge deployment 2500 using an edge heartbeat component in an edge host, according to some aspects. Referring to FIG. 25 , edge deployment 2500 includes a service coordinating entity, such as, for example, an MEC host 2502, coupled to multiple base stations 2504, 2506, ..., 2508. The base stations can each be coupled to multiple edge devices, such as, for example, 2510, ..., 2512, 2514, ..., 2516, and 2518. (Although an MEC host is illustrated, the concepts are equally applicable to fixed or non-mobile edge computing nodes.)

[0376] In some examples, each edge device 2510, ..., 2512, 2514, ..., 2516, and 2518 may be communicatively coupled to the MEC host 2502 via a direct communication link (e.g., the dashed line shown in FIG. 25 ) or via a corresponding base station. In some aspects, each edge device may be coupled to one or more other edge devices that use the MEC service 2520 via multiple communication links (e.g., communication link 2536 used by device 2514). In this regard, information, including heartbeat information, can be passed from device to device to be communicated back to the MEC host 2502 for use by the edge heartbeat component 2522.

[0377] In some examples, the MEC host 2502 runs an edge service, such as, for example, an MEC service 2520. The MEC service 2520 can be communicatively coupled to exchange information with the base stations 2504, 2506, 2508 and the edge devices 2510, ..., 2512, 2514, ..., 2516, and 2518. The MEC host 2502 further includes an edge heartbeat component 2522, which can have suitable circuitry, interfaces, and / or instructions configured to perform one or more functions described herein related to connectivity monitoring and connectivity assurance. Such functions may include: constructing a device heartbeat, which may include information indicating device availability for communication with the MEC service 2520 (e.g., whether the device is turned on and online / available for communication), information indicating device location (e.g., location relative to the base station to which the device is coupled, or location relative to the MEC host 2502), or other device-related information; propagating location information received as part of the heartbeat information (e.g., location information relative to the corresponding base station coupled to the device) to the base station (or other device); tracking whether an edge device is down (e.g., turned off or otherwise unavailable for communication); recording devices that are available (or unavailable) for communication with the MEC service; implementing a geofencing mechanism; etc.

[0378] In some aspects, the MEC host 2502 is configured (e.g., by a mobile network operator) to implement heartbeat reporting by clusters of edge devices (e.g., via heartbeat configuration information 2524). In one example, each cluster is configured (e.g., via heartbeat configuration information 2524) with heartbeat frequency, heartbeat information specifications, etc. In some aspects, the edge heartbeat component 2522 is configured with one or more threshold parameters (e.g., threshold K described in connection with FIGS. 23 and 24) to determine when to trigger a notification (e.g., to a mobile operator) that the MEC service is over- or under-subscribed / used.

[0379] After the MEC host 2502 communicates the heartbeat configuration information 2524 to the edge devices, the heartbeat information can be periodically reported back to the MEC host. For example, the edge devices 2514, ..., 2516 can report corresponding heartbeat information 2532, ..., 2534. In some examples, each edge device includes one or more hardware and / or software components (e.g., within a network interface card) for implementing heartbeat reporting functionality, geofencing functionality, etc. For example, the edge device 2518 includes a network interface card 2528 with a heartbeat component 2530 configured to report the heartbeat information as configured by the MEC host 2502.

[0380] In some aspects, the edge heartbeat component 2522 in the MEC host 2502 can configure one or more geofences using the geofence configuration information 2526. For example, the edge heartbeat component 2522 can use the geofence configuration information 2526 to configure a first geofence 2538, which can be associated with the location of the base station 2504, such that only devices associated with the base station 2504 and within the geofence 2538 can use the MEC service 2520. The edge heartbeat component 2522 can configure additional geofences, such as geofence 2540, based on the locations of other edge devices (e.g., base station 2506), which can be used to configure edge devices within the geofence to use the MEC service 2520. In some aspects, geofences can be configured based on other criteria, such as proximity to the MEC host 2502. In yet a further example, an LSM or other security policy can describe the geofence policy, and the MEC host, tower, base station, or other node with coordinates and geofences can act as an LSM enforcement point.

[0381] In a further example, attesting location can enable the RoT capabilities of a location sensor to establish greater confidence in location coordinates. Thus, global navigation (e.g., GPS) coordinates can be reported conditional on satellite signals, where the satellite also contains the RoT key and attestation signal, providing greater reliability and resilience to geofence access enforcement mechanisms.

[0382] Although the edge heartbeat component 2522 is shown as being implemented within the MEC host 2502, the disclosure is not limited in this respect and the edge heartbeat component may be implemented within one or more other edge devices, such as, for example, within one or more of the base stations shown in FIG. 25.

[0383] In some examples, each base station (2504, ..., 2508) or MEC host 2502 can be configured to implement a heartbeat or beacon for a group of edge devices, including setting the frequency of reporting and thresholds that can be used to trigger alarms.

[0384] In some examples, the edge heartbeat component 2522 may be configured to use the received heartbeat information to track the following information related to edge devices in the edge deployment 2500: device ID, base station ID (for a base station associated with the identified device), central office (CO) ID (for a CO associated with the identified device), connected device ID list (e.g., a list of IDs of edge devices connected to the identified device), etc. In some aspects, the edge heartbeat component 2522 may use one or more Bloom filters (or other software or hardware structures) to implement such tracking mechanisms and to assist in performing one or more of the heartbeat processing functions in connection with connectivity evaluation.

[0385] In some examples, the techniques disclosed herein can be applied in an edge / MEC environment context (e.g., such as implemented in accordance with the MEC / 5G architectures shown and described in Figures 75-78) to ensure that a system of edge nodes spanning the gap between an access edge and a cloud edge ensures that a path (or multiple paths) exists from edge to edge. Such functionality prevents the possibility that a service or resource disruption (e.g., brownout) in an edge cloud instance (e.g., the edge cloud 110 and edge computing system configurations shown in Figures 3 through 22D) will not be proactively detected.

[0386] In some examples, heartbeat reporting can be configured such that each device periodically reports heartbeat information at a predetermined frequency. In some examples, each base station or MEC host (e.g., a device or devices configured with an edge heartbeat component) can periodically poll (or request) heartbeat information from each edge device in the edge network.

[0387] In a further example, a method related to a technique for managing device connectivity of multiple edge nodes in an edge computing network may be implemented using an edge heartbeat component. In some aspects, the method may be performed by an edge heartbeat component (e.g., 2522) of a service coordinating entity (e.g., MEC host 2502) executing an edge service (e.g., MEC service 2520) that communicates with multiple edge nodes (e.g., 2510, ..., 2518) in the edge computing network. In a first operation, heartbeat configuration information (e.g., 2524) is encoded for transmission to the multiple edge nodes, the heartbeat configuration information configuring reporting of device heartbeat information. In a second operation, device heartbeat information received from the multiple edge nodes is decoded based on the heartbeat configuration information. In a third operation, the number of edge nodes among the multiple edge nodes connected to the computing service is determined based on the device heartbeat information. In a fourth operation, a notification indicating device connectivity to the computing service is generated based on the determined number of edge nodes. For example, an alert may be generated if it is determined that the number of edge nodes using the edge service is greater than a threshold number. Other types of alerts or notifications may also be generated based on the received heartbeat information to manage device connectivity within the edge computing network and assist in service guarantees.

[0388] A first example (Example A1) of a method for implementing edge heartbeats is a method for managing device connectivity of multiple edge nodes in an edge computing network (e.g., implementing systems and devices such as those implemented on edge cloud 110 and nodes or devices 2200, 2232, 2240, or 2250), the method including: by one or more processors of a service coordinating entity executing a computing service that communicates with multiple edge nodes in the edge computing network: encoding heartbeat configuration information for communication to the multiple edge nodes, the heartbeat configuration information configuring reporting of device heartbeat information; decoding device heartbeat information received from the multiple edge nodes based on the heartbeat configuration information; determining, based on the device heartbeat information, a number of edge nodes among the multiple edge nodes that are connected to the computing service; and generating a notification indicating device connectivity for the computing service based on the determined number of edge nodes.

[0389] In a second example (Example A2), the matter related to Example A1 includes, by one or more processors of the service coordinating entity: generating a notification when the number of edge nodes connected to the computing service is less than a threshold number.

[0390] In a third example (Example A3), the matter of Examples A1-A2 includes that the device heartbeat information includes device location information for each node of the plurality of edge nodes.

[0391] In a fourth example (Example A4), the matter related to Example A3 includes that the device location information includes location information for a service coordinating entity or for a base station communicatively coupled to multiple edge nodes and the service coordinating entity.

[0392] In a fifth example (Example A5), the matter of Example A4 includes the heartbeat configuration information configuring the frequency at which the device heartbeat information is reported.

[0393] In a sixth example (Example A6), the matters related to Examples A1-A5 include that the device heartbeat information received from at least one edge node of the plurality of edge nodes includes a data packet indicating a transmission time of a data packet transmitted by the at least one edge node.

[0394] In a seventh example (Example A7), the matter related to Example A6 includes, by one or more processors of the service coordinating entity: determining a communication latency between the at least one edge node and the service coordinating entity based on the transmission time and the time the data packet was received by the service coordinating entity; and generating a notification based at least on the determined communication latency.

[0395] In an eighth example (Example A8), the matter of Examples A3-A7 includes, by one or more processors of the service coordinating entity: configuring a geofence associated with the computing service.

[0396] In a ninth example (Example A9), the matter of Example A8 includes, by one or more processors of the service coordinating entity: determining, based on device heartbeat information, a number of edge nodes among the plurality of edge nodes that are outside the configured geofence; and generating a notification based at least on the determined number of edge nodes that are outside the configured geofence.

[0397] In a tenth example (Example A10), the matter of Examples A1-A9 includes a service coordinating entity as a multi-access edge computing (MEC) host running a computing service as an MEC application instantiated on the virtualization infrastructure of the service coordinating entity.

[0398] In an eleventh example (Example A11), the matter of Example A10 includes that the MEC host is configured to operate in accordance with standards from the European Telecommunications Standards Institute (ETSI) MEC family of standards.

[0399] In various environments, Examples A1-A11 (and other aspects of this edge heartbeat use case) may be observed or monitored as a result of APIs or specifications that govern the use of heartbeats; protocols that govern or involve the use of heartbeats; and other uses and implementations of heartbeats in the following examples (e.g., Examples B1-AA11). Examples A1-A11 and other aspects of heartbeats may also be observed or implemented as a result of service operations and service functions (e.g., to be launched or operated in a FaaS or EaaS environment, such FaaS or EaaS elements may be comprised of code elements designed to run on compute elements within an edge cluster or to be serviced by some compute element in communication with the edge cluster that functions to achieve the SLA objectives and obligations or other obligations of the edge cluster). Furthermore, the methods of Examples A1-A11 may be provided to an edge computing system as implemented instructions (e.g., in a machine-readable medium that, when executed, performs the method) or as implemented hardware configurations (e.g., in an apparatus including configuration for performing or achieving the method). Thus, the features of Examples A1-A11 (and the heartbeat service) may be combined with any of the other examples herein to configure an edge cluster or edge cloud system, as coordinated or designed by a system orchestrator or architect.

[0400] Examples of traceable data movement

[0401] Edge deployments are highly dynamic, and therefore, one consideration in such deployments is service guarantees and data proof-of-concept. Edge computing is unique in that there can be tens or hundreds of thousands of network (or edge) devices (e.g., drones, sensors, user equipment (UE), or other computing devices) communicating information with edge services (e.g., computing services running on service coordinating entities, such as those described above with respect to the edge services and functions illustrated in and described in connection with FIGS. 7 through 12). Having a large number of network devices means that at any given instance, different types and amounts of information are exchanged between the devices. For example, sensors may continuously send different types of information to the edge. A key challenge that arises is the issue of trust: how can the edge infrastructure handle thousands of devices transmitting information while maintaining a means to trust the source of the data, which may flow through several intermediate hops on its way to the edge infrastructure?

[0402] The techniques disclosed herein can be used to handle data attestation in edge clouds (e.g., the edge cloud 110 and edge computing system configurations shown in FIGS. 3-22D). More specifically, the disclosed techniques can be used to provide edge devices and infrastructure with the hooks necessary to track the flow of events involving data transfer. For example, this can provide a trusted mechanism for sensor A to send data to gateway B to hub C to edge server D while simultaneously processing several other such streams (thus dynamically moving workload data to the relevant computational resources).

[0403] 26 illustrates an example of data flow between edge nodes (e.g., implementations of nodes 302, 312, 322) in accordance with some aspects. Referring to FIG. 26, illustrated is a data communication exchange between multiple edge nodes 2602 in an edge computing network 2600. More specifically, the edge nodes 2602 (which may include edge network devices AI) exchange data via data flow paths 2604 (illustrated as edges between the edge nodes 2602).

[0404] In some aspects, an edge lookup table may be cached at multiple network destination locations within the edge computing network 2600. The edge lookup table (LUT), which may be the same as the LUT 2728 of FIG. 27, may include authentication information for each of the edge nodes 2602. For example, the LUT may list authenticated edge nodes in the edge computing network 2600, authentication credentials for each of the edge nodes, and other authentication information that may be used for data authentication. In some aspects, the LUT may be provided to the edge nodes via a service coordinating entity, such as an MEC host, as shown in FIG. 27 (e.g., implemented in accordance with the MEC architecture shown and described in FIGs. 75-78).

[0405] In some aspects, the LUT can be used as a reference mechanism through which information is attributed to the source at each data communication hop. For example, when data is communicated between edge nodes, the data packet can include an edge handle that specifies, for example, the edge node that the data traversed. For example, a quantity of data can have an edge handle indicating that an edge node having ID A→B→C→D transmitted the packet, and the LUT can be used to determine whether one or more of the edge nodes specified by the edge handle are authorized for data exchange within the edge computing network 2600. In one example, each node in a flow path can be an LSM or other security policy enforcement point, particularly with the use of flow-oriented security policies or security functions.

[0406] If a node determines that it is not authenticated or that data attestation for a communication exchange indicated by an edge handle has failed, a notification / alert may be generated and communicated within the edge computing network. In a further example, information centric networking (ICN) functionality may be included or adapted for use within edge computing network 2600 or 2700.

[0407] 27 illustrates techniques for attesting data within an edge computing network, according to some aspects. Referring to FIG. 27, an edge computing network 2700 includes an edge computing host 2702 (e.g., an MEC host) coupled to a plurality of edge nodes (or edge devices) 2704, 2706, 2708, ..., 2710 (e.g., implementations of nodes 302, 312, 322).

[0408] In some aspects, the edge computing network 2700 may use mechanisms for trusted flow transmission, such as using hardware or software hooks, when data is communicated between edge nodes to ensure that data has not been tampered with and that participating edge nodes are authenticated for data exchange. In some aspects, at each network location (e.g., edge nodes 2704, ..., 2710) involved in data movement (e.g., receiving or sending data), including communications between multiple edges or edge devices, the data payload may be hashed and an optional timestamp may be included. For example, in some aspects, a data packet 2716 originating from the edge node 2704 may be hashed or encoded with a trusted network key (e.g., a trusted key that may be communicated to an authenticated edge node in the edge computing network 2700). As the data packet traverses to other edge devices (e.g., 2706, ..., 2710), each edge node can decode the data, add additional information (e.g., add the source edge device ID, the device IDs of the edge devices that have participated in the data flow so far, timestamp information when the data is received or transmitted at each hop, and other information to the edge handle attached to the data packet's data payload), re-encode the data packet, and forward it to the next device. In this regard, when the data packet arrives at an edge service (e.g., performed by edge compute host 2702) or target destination, the receiving edge node at each hop can check where the data packet came from, when the data packet was received or transmitted by each device in the data flow path, which devices are in the data flow path and whether the devices are authenticated for communication within edge computing network 2700, and other authorization and authentication processing functions for data authentication.In some aspects, data communicated with each data packet (eg, an edge handle) may be used for debugging purposes as messages are forwarded.

[0409] In some aspects, as illustrated in Figure 27, an edge handle may be added to the data payload to form a data packet as the packet travels from a source edge device to a destination device. For example, if device B receives data authenticated by edge device A, device B may authenticate itself and include A as the data source, packaging this information with an edge handle to form a data packet that is forwarded to edge device C. Edge device C then authenticates the flow from A to B to C, generates an updated edge handle, and forwards the data packet to edge device D.

[0410] In some aspects, the certification / authentication process of the edge device may be based on blockchain technology or other types of tracking or authentication technology. In some aspects, each edge handle may include a source edge device ID and a data flow path (e.g., device identification information for each device that participated or is participating in communicating the data packet). In some aspects, a timestamp may also be included for one or more of the indicated devices participating in the data flow path (e.g., the timestamp indicates when data is sent from or received at the corresponding edge device that is part of the data flow path).

[0411] 27, data packet 2716 originates from and is generated by source edge node 2704. Data packet 2716 includes an edge handle 2715 that is appended to a data payload to form data packet 2716. Edge handle 2715 (i.e., EH1) can include information 2722, such as the edge device ID of the originating (source) edge device (e.g., the identifier of edge device 2704, which is the source device of data packet 2716), as well as the device identifiers of the edge devices that form the current data flow path for the data packet. Because this device is the source device, the data path in edge handle 2715 includes the identification information of only node 2704.

[0412] After data packet 2716 is generated, edge node 2704 transmits the data packet to the next device in the communication path, such as edge node 2706. After the packet is received at edge node 2706, edge node 2706 updates the edge handle 2724 (e.g., to EH2) by updating the data flow path to include the identification of edge node 2706. For example, the data flow path in edge handle EH2 may indicate "AB," indicating that the data packet originated at edge device A (2704) and was received at edge device B (2706). After edge node 2706 updates the edge handle to generate EH2, EH2 is appended to the data payload to form a new data packet 2718, which is communicated to the subsequent edge device in the data flow path, such as node 2708.

[0413] After the packet is received at edge node 2708, the node updates the edge handle 2726 (e.g., to EH3) by updating the data flow path to include the identity of edge device 2708. For example, the data flow path in edge handle EH3 may indicate "AC," indicating that the data packet originated at edge node A (2704), was forwarded by edge node B (2706), and was received at edge node C (2708). After edge node 2708 updates the edge handle to generate EH3, EH3 is appended to the data payload to form a new data packet 2720, which is communicated to the subsequent edge device in the data flow path, such as device 2710. Additionally, an auditor, compliance entity, or third party may sign the EH value according to a publish-subscribe or other similar distributed messaging system so that data flows, attestation flows, or other flow graph activity can be monitored, analyzed, and inferred as edge telemetry or metadata.

[0414] In some examples, and as described above, when each edge device updates its edge handle, the edge handle may similarly include timestamp information. To ensure synchronized clock data between edge devices in edge computing network 2700, edge compute host 2702 may be configured with synchronization component 2714. Synchronization component 2714 may have suitable circuitry, interfaces, and / or instructions and may be configured to communicate a synchronization signal (e.g., via a broadcast message to all edge devices authorized to communicate within edge computing network 2700) to facilitate clock synchronization between edge devices.

[0415] In some examples, edge computing host 2702 is configured with an edge LUT component 2712. Edge LUT component 2712 may have suitable circuitry, interfaces, and / or instructions configured to generate and maintain a LUT (e.g., as described in connection with FIG. 26 ) that may be populated to all edge devices (e.g., via a broadcast message to devices 2704, 2710) in edge computing network 2700 so that the edge devices can use such LUT for device authentication and data attestation purposes.

[0416] In a further example, a method for implementing a technique for data attestation may be implemented within an edge computing network. The technique may be executed by one or more processors of a first edge device (e.g., 2706) of a plurality of edge devices in the edge computing network (e.g., 2700). In a first operation, a data packet (e.g., 2716) received from a second edge device (e.g., 2704) of the plurality of edge devices is decoded. The data packet includes a data payload and a first edge handle (e.g., EH1). The first edge handle indicates a source edge device (e.g., device 2704 or A) and a data flow path associated with the data payload. In a second operation, a second edge handle (e.g., EH2) is generated to include the source edge device (e.g., A) and an updated data flow path (e.g., edge device identifiers for devices 2704 and 2706). In this regard, the updated data flow path in EH2 is based on the data flow path (e.g., the data flow path pointed to by EH1, which is only device A) and the device ID of the first edge device (i.e., device B). In a third operation, the second edge handle and data payload are encoded (e.g., as data packet 2718) for transmission to a third edge device (e.g., 2708) of the plurality of edge devices.

[0417] As illustrated by these examples, within an edge cloud there will be many data providers, along with many edge computation services (e.g., the edge services and functions shown and described in connection with Figures 7-12) that perform transformations on data from the data providers. To enable a robust service environment, endpoint devices (and edge nodes) will need access to the transformed data, along with the ability to verify (attest) both the data providers and the edge services that are processing the data.

[0418] In one example, data and data transformation attestation is enabled within an edge computing system for the results or use of edge computational services. For example, when a data provider sends authenticated data to an edge service, the edge service can attest that the data originated from a trusted source. When the edge service applies a transformation (that the edge service is able to perform and has the ability or authority to perform), the results of the edge service are authenticated. As a result, data obtained from the edge service can be consumed by an endpoint device while verifying both the source of the data and the transformation of the data.

[0419] As a simple example, consider a scenario where a camera from a vehicle transmits images to a cell tower or other wireless access point, and an edge service performs low-latency vehicle or object detection processing on the images. When users of this data receive the object detection results, they can verify that the images were provided by a trusted camera and, in addition, that the object detection analysis was performed by a trusted entity.

[0420] In some situations, the origin of the camera or data may be unknown, as the data may simply appear at an intermediate processing stage / node within the edge computing system. This processing includes attestation context when an intermediate node (prior to the current node context) indicates that the data has unknown origin. Thus, attestation information may include a combination of endpoints and the path connecting the endpoints when visual data is transcoded / translated through multiple hops.

[0421] The use of data and data transformation proofs may be enabled through interoperable proof data, including data prescribed by standards or specifications. Key to a proof infrastructure is the ability of a verifier (one or more) to discover resources it interacts with indirectly (as well as directly) (e.g., because each data transformation action may have involved a different set of nodes and transformation logic). Proofs of intermediate actions may be stored and made available for subsequent queries. For example, data and code "trust" may be represented as cryptographic hashes. A hash tree of attested resources may be used to maintain the current "accepted" proof value of all its predecessors. A centralized approach maintains hash tree updates and replicates a query-optimized copy as a "proof cache." A decentralized approach may utilize a blockchain.

[0422] In the context of data and data transformation attestation, a rich dictionary of "provables" and properties can be defined. This dictionary list can be as simple as a set of tag-value pairs, where the tag name identifies the provable and the value is its "measurement" (e.g., a hash). While a predefined list can include some of these values, user communities can also specify tags for resources for specific applications that require attestation. For example, a security camera owned by a stakeholder in an edge computing system can identify metadata (e.g., EXIF) about the images, while the camera manufacturer or vendor can issue a certificate containing information about the test suites that the camera has successfully passed.

[0423] Timestamps used to track data may require synchronized clocks, which can be a challenge for distributed systems. However, for tracking purposes, it may be sufficient to capture the sequence of events without capturing their timing. Because blockchain imposes ordering (the blockchain order may differ from the actual order, but the use case may not require actual precise ordering), blockchain is one way to achieve tracking without timing. In using ordered tracking, orchestration or workflow plans may be used as the metric because they describe the intended ordering. However, in simpler examples, where the focus of the use case is forensic purposes (e.g., to resolve disputes or characterize the pathology of an attack), using the local platform clock may be sufficient (especially if prerequisites are met, such as, for example, changes to the local clock are recorded on the blockchain and data flow to and from the platform is recorded on the blockchain).

[0424] A first example (Example B1) of a method for implementing traceable data movement is a method for data attestation within an edge computing network (e.g., implementing systems and devices such as those implemented on edge cloud 110 and nodes or devices 2200, 2232, 2240, or 2250), the method including: by one or more processors of a first edge device of a plurality of edge devices in the edge computing network: decoding a data packet received from a second edge device of the plurality of edge devices, the data packet including a data payload and a first edge handle, the first edge handle indicating a source edge device and a data flow path associated with the data payload; generating a second edge handle to include the source edge device and an updated data flow path, the updated data flow path based on the data flow path and the device ID of the first edge device; and encoding the second edge handle and the data payload for transmission to a third edge device of the plurality of edge devices.

[0425] In a second example (Example B2), the matters pertaining to Example B1 include a source edge device initiating communication of a data payload, and a data flow path indicating a subset of the multiple edge devices to be used to communicate the data payload.

[0426] In a third example (Example B3), the matter related to Example B2 includes, by one or more processors of the first edge device: using the data flow path to determine a device ID for each edge device in the subset of the plurality of edge devices.

[0427] In a fourth example (Example B4), the matters related to Example B3 include, by one or more processors of the first edge device: authenticating the subset of the multiple edge devices included in the data flow path based on the determined device ID and a device authentication lookup table.

[0428] In a fifth example (Example B5), the matter of Example B4 includes, by one or more processors of the first edge device: generating a notification when at least one edge device of the subset of the plurality of edge devices fails authentication.

[0429] In a sixth example (Example B6), the matters related to Examples B4-B5 include, by one or more processors of the first edge device: decoding a device broadcast message from a service coordinating entity in the edge computing network, and the device broadcast message updating a device authentication lookup table.

[0430] In a seventh example (Example B7), the matter of Example B6 includes that the device broadcast message further includes at least one authentication key for secure communication between multiple edge devices in the edge computing network.

[0431] In an eighth example (Example B8), the matter of Example B7 includes that the encoding and decoding are based on at least one authentication key.

[0432] In a ninth example (Example B9), the matters relating to Examples B6-B8 include, by one or more processors of the first edge device: synchronizing a device clock of the first edge device based on a clock synchronization signal in the device broadcast message.

[0433] In a tenth example (Example B10), the matter related to Example B9 includes, by one or more processors of the first edge device: encoding the second edge handle to further include a timestamp of the first edge device, the timestamp indicating the time when the data payload is transmitted to the third edge device.

[0434] In an eleventh example (Example B11), the matter related to Examples B6-B10 includes a configuration in which the service coordinating entity is a multi-access edge computing (MEC) host that runs computing services as MEC applications instantiated on the virtualization infrastructure of the service coordinating entity.

[0435] In a twelfth example (Example B12), the matter related to Example B11 includes an arrangement in which the MEC host is configured to operate in accordance with standards from the European Telecommunications Standards Institute (ETSI) MEC standardization family.

[0436] In a thirteenth example (Example B13), the matter relating to Examples B1-B12 includes a configuration that enables the results of an edge service in an edge computing network to be provable based on verifiable proof of the source of data provided to the edge service and the results of that data generated by the edge service.

[0437] In various environments, Examples B1-B13 (and other aspects of this trackable data movement use case) may be observed or monitored as a result of APIs or specifications that define data tracking configurations; protocols that define or involve data tracking configurations; and other uses and implementations of trackable data within an edge computing environment. Examples B1-B13 and other aspects of data tracking may also be observed or implemented from service operations and service functions (e.g., as invoked or operated in a FaaS or EaaS environment). Furthermore, the methods of Examples B1-B13 may be provided to an edge computing system as implemented instructions (e.g., in a machine-readable medium that performs the method when the instructions are executed) or as implemented hardware configurations (e.g., in an apparatus including configurations for performing or achieving the method). Thus, the features of Examples B1-B13 (and other features of trackable data movement management) may be combined with any of the other examples herein to configure an edge cluster or edge cloud system, as coordinated or designed by a system orchestrator or architect.

[0438] Globally shared / subscribable distributed edge queue example

[0439] Various forms of queue handling may be used within an edge computing platform (e.g., the edge cloud 110 and edge computing system configurations shown in FIGS. 3 through 22D) to coordinate and orchestrate the orchestration and service operations described herein. Various types of computing systems provide queues for storing jobs or functions to be performed. The concept of queues can be extended for use in a distributed edge computing platform to operate a global (e.g., multi-edge system) set of queues shared among multiple edge locations (e.g., base stations or other edge gateway nodes 312, edge aggregation nodes 322, etc.).

[0440] In one example, individual queues are kept consistent across different edge nodes, but the job definition (function or service to be executed, payload, and possibly data) is stored at the edge location where the request is submitted. Then, when a service request or function is requested by an edge device, the orchestrator at the edge location to which that device is connected adds the job to be executed to a queue (continuously), along with the metadata used (e.g., job definition, submitting edge node, etc.). Different orchestration entities running at each edge location pick up the job to be executed on the local edge node. Once the job is executed, the job (service or FaaS) is removed from the queue (continuously), and the data and job definition can be moved and executed as needed.

[0441] The use of shared or subscribeable edge queues can support the concept of workload balancing in a distributed environment. A structured method can be envisioned for the use of queues, where local queue details are known locally only to the corresponding orchestrator (e.g., MEC orchestrator, MEO), and only queue sizes are exchanged between orchestrators (e.g., MEO) at time intervals. Using this information, the goal of reducing the occurrence of queue overflows (and equivalently, job "drops") can be attempted.

[0442] Various device implementations of subscribable distributed edge queues may be provided within a service orchestration and execution system to be deployed across any number of edge computing hardware, circuitry, nodes, or system configurations (e.g., the edge cloud 110 and edge computing system configurations shown in FIGS. 3 through 22D). Various example methods may include adding, adjusting, and removing information from globally accessible queues and executing jobs (e.g., one or more services, service actions, workflows, etc.) according to coordinated use of the queues. Thus, methods executed on various processing circuits of edge computing system nodes may include deploying and using shared queues within the edge computing system to coordinate the orchestrated execution of jobs and functions across nodes.

[0443] In various environments, globally shared or subscribeable queues may be observed or monitored as a result of APIs or specifications that govern the use of data tracking configurations; observations through traffic analysis within the network; and other uses and implementations of queues and queue data within edge computing environments. Globally shared or subscribeable queues may also be invoked, observed, or implemented as a result of service operations and service functions within edge computing systems (e.g., to support sharing between services in a FaaS or EaaS environment). Thus, features of the above examples (and other features of distributed queues) may be combined with any of the other examples herein to configure an edge cluster or edge cloud system, as coordinated or designed by a system orchestrator or architect.

[0444] Automated Edge Service Distribution Example

[0445] As suggested by many of the examples above, an edge cloud architecture consists of multiple edges (e.g., small cells, cell towers, or different points of aggregation) that can be grouped into different edge-cooperative clusters. For example, these can be defined by the infrastructure owner. Some of the other systems and techniques described throughout this document discuss methods for resource definition, discovery, and security grouping to achieve various goals. The systems and methods described below can provide service distribution for these groups. The set of attributes that are common (e.g., across group members) is one way of describing grouping semantics. A discovery process can query a group's attributes and compare overlap with a set of attributes of interest as one way to determine collaboration suitability. Attributes, in one example, can be captured as a set of interoperable, semantically rich tags. The tag namespace can be extensible, and collaboration orchestration can be used to discover tags that are technically different but semantically the same. Tags can also have values ​​(e.g., CODEC can be a tag, and the value can list supported CODEC algorithms).

[0446] The attribute tagging architecture can be scalable to multi-tenant collaborative edge architectures (e.g., the edge cloud 110 and edge computing system configurations shown in FIGS. 3 through 22D). The attribute tagging architecture can be implemented using hardware that provides low latency and efficiency, which may be needed to provide quick turnaround and improve overall cost of ownership (e.g., as few resources as possible used only when truly needed). Hardware-based implementations can also provide high computational and acceleration density in limited space with power and thermal constraints, as is the case for edge cloud solutions located within base stations. Existing solutions that may be suitable for traditional data centers are not suitable for such environments.

[0447] The grouping architecture can extend existing platforms (or create new platforms) where accelerated features or feature requests are handled directly by the platform; therefore, no software is required. This can result in less maintenance (a consideration in edge systems where large numbers (e.g., 100K+) of base stations are deployed), greater compute execution density (e.g., better TCO), or better response times by accelerating request processing from clients. Furthermore, such an approach can provide a mechanism for bringing (e.g., moving, federating, or mobilizing) compute resources to workload data so that workload data can also be brought (e.g., moving, federating, or mobilizing) compute resources.

[0448] Edge cloud architectures are emerging as one of the promising areas where edge network and data center architectures have the opportunity to enable new use cases not previously possible. In this context, as shown, one meaningful use case for telecommunications service providers (TSPs) and cloud service providers (CSPs) is exposing Function-as-a-Service (FaaS) (e.g., processing an image and returning an object) to applications running on edge devices (e.g., smartphones or IoT) to accelerate their workloads and applications. In this context, acceleration Function-as-a-Service (similar to FaaS where the function is implemented and executed in an accelerator) can be used, as described below.

[0449] As mentioned above in the overview of edge cloud deployment, accelerators and compute are placed in base stations (including MEC platform configurations) to deliver low latency FaaS and AFaaS (1-10 ms latency). However, one of the main challenges to placing processors and accelerators in base stations to implement scalable edge cloud FaaS and AFaaS is (1) physical space constraints, (2) low latency and scalable scheduling solutions for handling client requests / functions, and (3) high effective compute density and acceleration (the compute and acceleration used for the actual computation, not by the system software stack).

[0450] The systems and methods described herein involve a new type of MEC platform architecture (which can be applied to current systems and architectures) that allows a CPE (Customer Premises Equipment), small cell, or cell tower to redirect the execution of a particular service to other underutilized edges within the edge architecture while maintaining the QoS or SLA for that service.

[0451] Referring to Figure 28, the edge architecture consists of multiple edges (e.g., small cells, cell towers, or different aggregation points) that are grouped into different edge collaborative clusters (e.g., as may be defined by the infrastructure owner). Each individual in the cluster can broadcast (e.g., at configurable intervals) to other peers. The broadcast can include current utilization or future needs for different types of resources (e.g., storage, memory, etc.).

[0452] When a given edge location receives a request from an edge device 2802 (e.g., a smartphone or a car) to perform a given service with a given service level agreement (e.g., regarding deadline dates and resource requirements), the given edge location may decide to forward the request to another edge location. The forwarding decision may be based on one or more (non-exclusive) of the following factors:

[0453] Another edge location can perform the service given the other location's distance from the current edge (e.g., latency) and available resources. Utilization or predicted utilization at the current edge. For example, the edge location can include a small ML or DL ​​training computation element used to learn and predict upcoming requests. In one example, the selection of another location can be made using a different scheme (e.g., round robin, less utilized edge, monetary or resource cost considerations, etc.). The architecture may include a programmable logic device (e.g., implemented in an FPGA) that is responsible for selecting the forwarder and may be implemented by each infrastructure owner.

[0454] The forwarding edge architecture can include the use of a set of new services and business cases that are not available using current technology. The forwarding edge architecture can be applied to a wide variety of segments and workloads. Figure 20, for example, shows some examples that can be applied to the forwarding edge architecture.

[0455] FIG. 28 more specifically illustrates a system diagram involving an edge computing architecture 2800 for edge service distribution. As shown in FIG. 28, architecture 2800 can include and execute instructions (e.g., implemented via logic or code) to perform functions at individual edge locations 2804 (e.g., from customer premises equipment, small cells such as 2806, cell towers such as 2808, and different aggregation points such as a local central office 2812). The instructions can be provided within an edge gateway or platform 2810 for an entity and implemented via an intelligent NIC 2820 (also known as a “smart NIC”) or any other type of network card. These functions can include elements, such as those described below. In one example, functions can be applied using a tag or multiple tags, such as interoperable, semantically rich tags, which can also have values.

[0456] The functionality may include a set of new out-of-band interfaces 2822 that allow an infrastructure owner to register, for a given edge, one or more (or all) different edge peers that belong to the same collaborative edge cluster. In one example, an edge may belong to multiple edge clusters.

[0457] The capabilities may include data including the type of telemetry and resource utilization data 2824 to be broadcast to edge peer clusters. The telemetry may be different for each edge cluster. The capabilities may include the frequency at which an edge broadcasts telemetry and resource utilization data 2824 to the edge collaborative cluster to which it belongs. This telemetry may be specific (or shared) to each owner, tenant, or service.

[0458] The function may include a bitstream (or the manner in which the acceleration algorithm 2826 is implemented) that is responsible for selecting different edges or edges in different registered clusters (if any). The registered clusters may correspond to the edges where the service request resides. This bitstream may have access to telemetry data tables to make such decisions.

[0459] The function may include a telemetry data structure responsible for storing data coming from different edge peers. In one example, the data may be organized into different edge collaborative clusters. This function may use machine learning on the types of service submissions arriving at a given edge to predict future utilization of that edge. This machine learning technique may be implemented in programmable logic devices as well.

[0460] A function is responsible for processing requests coming from individual or multiple edge users, tenants, and owners. Requests may come with a service to be performed or with SLAs and source requirements. Requests may be scheduled by the aforementioned bitstream. In one example, distributed telemetry data may be used to collect telemetry data and resource utilization configured by the infrastructure owner and broadcast the data to the different clusters it is registered to.

[0461] In a further example, a method for edge service distribution can include the following acts: the method includes a first act of receiving a request at a first edge location belonging to an edge cluster; the method includes a second act of selecting a second edge location of the edge cluster; selecting the second edge location can include using a tag of the second edge location, the tag indicating available resources and membership of the second edge location in the edge cluster.

[0462] The method includes a third operation of determining, at the first edge location, that the request can be performed at a second edge location of the edge cluster based on available resources at the second edge location. In one example, determining that the request can be performed at the second edge location includes determining that performing the request at the second edge location requires fewer computational resources or less time than performing the request at the first edge location (e.g., it is more efficient to perform the request at the second edge location than at the first edge location). The efficiency may be based on a predicted utilization rate of the first edge location over a period of time or may be based on a distance from the second edge location to the first edge location. The efficiency may be determined using a machine learning element to predict requests coming to the first edge location.

[0463] The method provides a fourth operation for forwarding the request to a second edge location for execution. The method may further include receiving, at the first edge location, telemetry and resource utilization data from the second edge location.

[0464] A first exemplary method (embodiment C1) for implementing edge service distribution is a method performed using processing circuitry (e.g., a processing circuitry of a node or device 2200, 2232, 2240, or 2250). The method includes receiving a request at a first edge location belonging to an edge cluster, determining at the first edge location that the request can be executed at a second edge location of the edge cluster based on available resources at the second edge location, and forwarding the request to the second edge location for execution.

[0465] In a second example (Example C2), the subject matter of Example C1 includes determining that the request can be performed at the second edge location by determining that performing the request at the second edge location requires fewer computational resources or requires less time than performing the request at the first edge location.

[0466] In a third example (embodiment C3), the technical matter of embodiment C2 includes determining whether executing the request at the second edge location requires fewer computing resources or less time based on the predicted utilization of the first edge location over a period of time.

[0467] In a fourth example (embodiment C4), the technical details of embodiments C2-C3 include determining whether executing the request at the second edge location requires fewer computing resources or less time based on the distance from the second edge location to the first edge location.

[0468] In a fifth example (Example C5), the technical matter of Examples C2-C4 includes determining that executing the request at the second edge location requires fewer computing resources or less time includes using a machine learning element to predict requests coming to the first edge location, where the machine learning element is trained based on historical requests.

[0469] In a sixth example (Example C6), the technical details of Examples C1-C5 include selecting a second edge location using a tag of the second edge location, the tag indicating available resources and the membership of the second edge location within an edge cluster.

[0470] In a seventh example (embodiment C7), the technical matter of embodiments C1-C6 includes receiving, at the first edge location, telemetry and resource utilization data from the second edge location.

[0471] In various settings, examples C1-C7 (and other aspects of automated edge service distribution) may be observed or monitored as a result of APIs or specifications defining the use of service distribution, protocols defining or encompassing the use of service distribution, and other uses and implementations of services in an edge computing environment. Examples C1-C7 and other aspects of edge service distribution may also be observed or implemented as a result of coordinated service operations and service functions (e.g., to enable coordinated services to be invoked or operated in a FaaS or EaaS setting). Additionally, the methods of examples C1-C7 may be provided in an edge computing system as implemented instructions (e.g., using a machine-readable medium that, when executed, performs the method) or as implemented hardware configurations (e.g., using an apparatus including configurations for performing or achieving the method). Thus, the features of examples C1-C7 (and other features of automated edge service distribution) may be coordinated or designed by a system orchestrator or architect to combine with any of the other examples herein to configure an edge cluster or edge cloud system.

[0472] Examples of Acceleration Execution and Definition

[0473] In one example, edge computing scenarios can be adapted to consider and operate acceleration for services and hardware in distributed computing environments. In traditional settings, acceleration is typically applied within local machines with fixed function-to-accelerator mappings. For example, in traditional approaches, a given service workload may invoke a given accelerator, and if this workload needs to be moved or migrated to another location (based on changes in network access, to meet new latency requirements, etc.), the workload may need to be re-analyzed, and potentially re-analyzed.

[0474] The following adaptations of edge cloud architectures (e.g., adaptations of edge cloud 110 and edge computing system configurations shown in FIGS. 3 through 22D) enable the coordination of different acceleration functions on different hosts (e.g., edge computing nodes). Rather than processing accelerator workloads serially, a parallel distribution approach is used to "divide and conquer" workload and function operations. This parallel distribution approach can be applied across the use of the same or multiple forms of acceleration hardware (e.g., FPGAs, GPU arrays, AI accelerators) and the same or multiple types of workloads and invoked functions.

[0475] Acceleration distribution may be used when a client device wants to run acceleration functions on a workload that provides a large amount of data (e.g., 10 GB, or some other amount of significant size for the type of device or network). If this chunk of data supports parallel processing—where the data can be run or analyzed using multiple accelerators in parallel—acceleration distribution may be used to distribute and collect acceleration results from multiple processing nodes. The following acceleration distribution approach may also be used when a client device wants to run a large number of functions (e.g., 100 or more functions at once) that can be run in parallel to fulfill the workload more efficiently or in a timely manner. The client device submits the data and workload data to be run with a given SLA and given resource cost. The workload is distributed, federated, and collected among multiple processing nodes—each node offering a different flavor or permutation of acceleration.

[0476] Figure 29 illustrates an architecture for accelerating data in parallel within an edge computing system 2900. Although not depicted in this system, there may be a flavor gateway that is used to identify a given flavor configuration, obtain attestations from each flavor service, FaaS, or other member, and provide an aggregated attestation result (e.g., a "flavor attestation"). In a further scenario, a workload orchestrator may require a flavor attestation as a prerequisite for scheduling a workload for a particular flavor group. And, in a further scenario, workloads may also be accelerated in parallel edge deployments.

[0477] In a first example, a set of workloads 2920 (operating under an SLA or other service approach) is offered by one or more client mobile devices 2910 for processing in the edge cloud (e.g., as a result of services executed by edge computing system 2900 as provided to client mobile devices 2910). This set of workloads 2920 can invoke one or more acceleration functions for execution by the edge cloud. For example, workload set 2920 can include a first workload 2921 and a second workload 2922. The acceleration required to process the first workload 2921 can invoke a first function-as-service (FaaS-1) 2931. The acceleration required to process the second workload 2922 can invoke the first function-as-service in addition to a second function-as-service (FaaS-2) 2932.

[0478] Workload set 2920 is distributed among N processing nodes in parallel—because the N processing nodes include a combination of the same or different acceleration hardware. For example, assume that first node 2941 includes acceleration hardware (e.g., an FPGA) to support execution of first function 2951, while second node 2942 and third node 2943 include the same acceleration hardware. As a result, a request to invoke a first function, such as by function 2931 as a first service, may be satisfied by execution of a workload comprising the first function among nodes 2941, 2942, and 2943, while another function may be executed on node 2942.

[0479] In one example, the distribution of workload among processing nodes may be coordinated by the edge gateway 2940, an orchestrator (not shown), or another intermediate edge cloud node. For example, the edge gateway 2940 may be responsible for determining the number of accelerators to which the data needs to be allocated (or determining the functions to be invoked by the accelerators), selecting the nodes or accelerators on the nodes, and sending the data to the selected nodes and accelerators. When the accelerators complete processing of the data and return the data, the edge gateway 2940 may apply a reduction clause (e.g., sum all results) or perform other processing. In addition to providing the payload, the client device 2910 may also specify the function(s) and the reduction clause(s) to apply, or the edge gateway 2940 may automatically identify the function(s) and reduction clause(s). Workload may be allocated independently of the edge gateway 2940, such as a portion of the workload (second function 2952) being allocated to another node 2944.

[0480] In a further example, the orchestration and coordination of workloads distributed among multiple accelerators may include aspects related to load balancing, mobile access prediction, or the use of telemetry, using other techniques discussed herein. The orchestration and coordination of workloads distributed among multiple accelerators may also be based on SLA / SLO criteria and objectives, as accelerators are selected to be most likely to meet SLA / SLOs based on current or predicted execution conditions.

[0481] Additionally, in the environment of FIG. 29, the NIC can be an LSM or other security policy enforcement point. For example, the NIC can prevent a node from accessing the network if it is not properly configured, contains a virus, or does not perform required actions. This is similar to network access control (NAC) use cases in the industry. Additionally, in such a configuration, the orchestrator can enforce the LSM or other security policy on the NIC, as the NIC can allow remediation traffic while simultaneously blocking normal activity.

[0482] In various edge system configurations, acceleration capabilities can be further enhanced through the use of data-abstract acceleration workload descriptions. These acceleration workload descriptions can be made available within the edge infrastructure through the use of uniform workload definitions defined in high-level formats such as XML. Such acceleration workload descriptions can be utilized to enable uniform access to accelerated resources and enhanced use of acceleration across different nodes in the edge computing ecosystem.

[0483] As one example, as an endpoint device moves between different edge access points in the infrastructure, the device can submit a workload definition to the edge access point to invoke acceleration. Such a workload definition may include a workload definition, data payload, an indication of associated SLAs / SLOs for the workload, resource costs, and other parameters. The receiving edge access point (edge ​​computing node) selects an appropriate accelerator, converts the workload into a specific bitstream definition (compatible with the accelerator), and executes the workload.

[0484] The acceleration description approach can enable edge computing systems to take advantage of definitions provided in standardized IoT frameworks (e.g., Open Connectivity Foundation (OCF), OLE Process Control (OPC), Thing-Thing). Here, sensor / actuator / controller nodes are described using common information and data models (metadata). The behaviors for discovering, interacting, and obtaining results are also captured by the data models in the form of interface definitions. In this way, edge infrastructure with acceleration resources can be similarly abstracted. This provides a form of "hierarchical service layer," in which higher layers expose the data model abstractions of lower layers. However, unlike IoT framework abstractions, these acceleration definitions provide a "service hosting" abstraction. Service hosting differs from a "service provider" abstraction (e.g., as used in discovery approaches such as UPNP) because the services or functions to be invoked are not specified. Rather, acceleration definitions provide a description of the hosting environment characteristics and constraints.

[0485] Finally, the use of acceleration workload descriptions can be extended to enable the use of "trusted" hosting environments, where the security-enhancing properties of the environment are also exposed for discovery and consideration. This enables the use of attestation in edge computing environments, such as from the selection of trusted (certified) acceleration resources.

[0486] A first exemplary method (embodiment D1) for implementing data-abstracted acceleration is a method performed using processing circuitry (e.g., processing circuitry of a node or device 2200, 2232, 2240, or 2250). The method includes receiving, at an edge gateway node, a workload that invokes an acceleration function, distributing data of the workload from the edge gateway node to each edge computing node for execution in parallel by the acceleration function of each edge computing node, and collecting, at the edge gateway node, results of the distributed execution by the acceleration function.

[0487] In a second example (Example D2), the technical matter of Example D1 includes the same type of acceleration function used by the acceleration function of each edge computing node.

[0488] In a third example (embodiment D3), the technical matter of embodiment D2 includes a workload that additionally invokes execution of at least one other edge computing node by a second type of acceleration function.

[0489] In a fourth example (Example D4), the technical matters of Examples D1-D3 include the acceleration function of each edge computing node being performed by at least one of FPGA, ASIC, GPU array, AI accelerator, transcoding accelerator, or cryptographic accelerator hardware.

[0490] In a fifth example (Example D5), the technical matters of Examples D1-D4 include a workload, which is a workload set including a first workload that invokes a function as a first service and a second workload that invokes a function as a second service, and the functions as the first and second services are provided between each edge computing node.

[0491] In a sixth example (embodiment D6), the technical matters of embodiments D1-D5 include a workload received from an edge client node.

[0492] In a seventh example (embodiment D7), the technical matter of embodiment D6 includes an edge client node specifying a workload to be executed according to a given SLA and a given cost.

[0493] In an eighth example (embodiment D8), the technical matters of embodiments D6-D7 include providing a collection result of distributed execution from an edge gateway node to an edge client node by an acceleration function.

[0494] In a ninth example (embodiment D9), the technical matters of embodiments D1-D8 include a configuration in which acceleration functions in an edge computing system are indicated in respective acceleration workload descriptions.

[0495] In various settings, Examples D1-D9 (and other aspects of acceleration management) can be observed or monitored as a result of defined application programming interfaces or interface specifications, use of protocols or definitions for initiating, receiving, or controlling acceleration, and other uses and implementations of acceleration within an edge computing environment. Examples D1-D9 and other aspects of acceleration management can also be observed or implemented as a result of coordinated service operations and service functions (e.g., as acceleration resources are initiated or operated in a service within a FaaS or EaaS setting). Additionally, the methods of Examples D1-D9 can be provided as instructions to be implemented in an edge computing system (e.g., using a machine-readable medium that, when executed, performs the method) or as hardware configurations to be implemented (e.g., using an apparatus including configurations for performing or achieving the method). Thus, features of Examples D1-D9 (and other features of acceleration management) can be combined with any of the other examples herein to configure an edge cluster or edge cloud system, as coordinated or designed by a system orchestrator or architect.

[0496] Edge device NIC processing example

[0497] As noted elsewhere, edge computing refers to processing workloads closer to users, such as at base stations, cell towers, and so on. In environments with many Internet of Things (IoT) devices—such as smart cities, smart transportation, and so on—artificial intelligence (AI) inference becomes an increasingly important workload. In such inference workloads, an orchestrator (e.g., an edge gateway) can forward inference requests to different service providers or instances to perform the inference. Thus, the edge orchestrator often directs the request and selects a platform to send the request to. The selected platform (e.g., a service running on that platform) receives the request and processes it to generate a result that the platform returns. Here, the orchestrator provides end-to-end load balancing and request tracking. Edge use cases, however, often require time-critical processing, resulting in very low latency processing and inference tolerance. Therefore, load balancing may not be optimal for such time-sensitive tasks, resulting in a higher total cost of ownership (TCO) or the SLA being lost due to hardware oversizing to meet the SLA. This scenario can be exacerbated when there is variability in inference latency.

[0498] To address the real-time inference needs of edge workloads, hardware-assisted technologies may be used in edge computing systems (e.g., edge cloud 110 and the edge computing system configurations shown in FIGS. 3 through 22D) to complement the orchestrator's load balancing scheme through smart network interface controllers (NICs). Specifically, these intelligent NICs are AI service-aware. The platform's NIC is notified of AI services running on the platform and memory regions corresponding to the "inputs" to the AI ​​services. AI services and their input memory buffers are registered in the NIC during service startup and maintained in a lightweight data structure (e.g., a table). Furthermore, a synchronization protocol may be used between the platform NIC, switch, and orchestrator to identify AI service-aware NICs on the platform. This information may be used to build a global view of which services or models are deployed within which platforms. In an AI-aware NIC, when a platform intelligent NIC receives a request, the NIC can place the request directly into the service provider's or model's input memory buffer for faster processing. Additionally, using an interface to measure the size of input memory buffers, an AI-aware NIC can also determine how much input has not yet been processed by the service.

[0499] The above capabilities lead to several techniques. For example, an orchestrator can query the platform NIC to identify the current input load on a service to improve load balancing. Also, if the receiving platform has multiple instances of an AI model running—such as from different vendors or service providers—and the request can map to any of them (e.g., with respect to security or accepted vendors), the orchestrator can simply forward the request to the platform, and the intelligent NIC can select the appropriate (e.g., best-matching) service provider for the request based on the input memory buffer state. This enables hierarchical orchestration, saving time and alleviating the orchestrator from micromanaging.

[0500] In an MEC setting, for an inference task, its nature is known to the MEC orchestrator (MEO), which can choose to delegate the task to an MEC / edge host that has already performed inference of the same nature / problem / category (e.g., face recognition). In this way, the inference results are more reliable because they come from a host that is already a "deep learner" for the task.

[0501] AI-aware or smart NICs can be used in conjunction with rack switches. For example, rack switches can obtain a list of services running on platforms hosted by the rack. These switches can use the list to identify the platform and send requests based on it, querying the platform NIC to obtain service status, or continuously tracking where requests are forwarded and returned.

[0502] In one example, when a platform NIC receives a request, it can forward the request to another platform running the requested service or model if its local memory buffers are full or heavy. This feature specifically enables dynamic load balancing without orchestrator involvement. In one example, the platform NIC accepting the request can notify the orchestrator about the forwarded request.

[0503] In one example, an intelligent NIC can use system telemetry to predict how long it will take to process a request, for example, based on current input buffer sizes. This prediction can then be used to determine to which of several platforms the request can be forwarded, for example, to reduce overall processing latency.

[0504] AI-aware NICs offer several advantages over standard NICs in the above scenarios, such as reduced orchestration and load balancing overhead, which can be important as this overhead can become significant due to the increasing number of IoT devices and services deployed at the edge.

[0505] FIG. 30 illustrates an example of an edge computing architecture 3000 including components for improved edge cloud inference processing. As shown, an edge gateway 3004 is in communication with a server rack 3005. The edge gateway 3004 may include an orchestrator 3010 configured to coordinate operations or configurations among server nodes (such as the server rack 3005) and may be configured to execute attestation logic 3012 and load balancing logic 3014 to support such operations or configurations. The server rack 3005 includes a top switch 3007 (e.g., a rack backplane or intra-rack switch fabric) and one or more servers (e.g., blades), such as server 3006. The top switch 3007 includes three components that are also present in the intelligent NIC of the server 3006: a service registry 3001, a load balancing component 3002, and a prediction component 3003. In other examples, the switch 3007 may be located elsewhere (not at the top).

[0506] Service registry 3001 is circuitry configured to maintain a mapping of services running on the current node, on server 3006, and on other nodes (e.g., rack 3005). Service registry 3001 includes an interface (not shown) through which a platform on server 3006 can register and deregister AI services for server 3006. Registration and deregistration can be initiated by orchestrator 3010 (e.g., in edge gateway 3004), a local daemon on the platform that is part of orchestrator 3010, a watchdog local daemon in the platform, or the AI ​​service itself. In one example, service registry 3001 is configured to initiate or receive broadcasts from other service registries (e.g., on NICs from other nodes) to update service mappings.

[0507] The load balancing component 3002 is a circuit configured to receive input for an AI service and send the input into the corresponding service's input memory buffer. The load balancing component 3002 is configured to determine whether to add the received input to the AI ​​service's memory buffer or forward it to another instance of the AI ​​service on another node. This decision may be based on a policy, such as the current state of the memory buffer or input from the prediction component 3003. In one example, the load balancing component 3002 is configured to respond to queries from the orchestrator 3010 regarding the current state of the input memory buffer.

[0508] The prediction component 3003 is a circuit configured to use system telemetry and, optionally, machine learning to understand and predict when an input can be processed by an AI service. To this end, the prediction component 3003 is configured to monitor the state of input memory buffers, which are provided as inputs to a machine learning system to provide a rough estimate of processing time. These outputs can be used by the load balancing component 3002 when determining whether an input to a service needs to be accepted or forwarded to another node. In a further example, metadata from other edge nodes can be used to perform better predictions. For example, recent requests performed by services ...

Claims

1. An edge computing system, comprising: a communications circuit configured to transmit and receive communications to and from a plurality of edge computing nodes; processing circuitry; The processing circuitry includes: Encrypting data related to the workload in a local data store in the source cloudlet to create encrypted data; Transferring a key for the encrypted data from the source cloudlet to a destination cloudlet; Receive workload migration instructions; and configured to, in response to the workload migration indication, transmit the encrypted data to the destination cloudlet over an insecure channel without re-encrypting the encrypted data for network transmission. Edge computing system.

2. The edge computing system of claim 1 , wherein the encrypted data is transferred from the local data store to a working memory as encrypted data.

3. 3. The edge computing system of claim 2, wherein the source cloudlet decrypts the encrypted data using full-memory encryption circuitry as the data is transferred from the working memory to a processor.

4. 10. The edge computing system of claim 1, wherein a portion of the encrypted data is not transferred as part of the workload migration, the portion being determined by profiling data usage by the workload.

5. The edge computing system of claim 1 , wherein the edge computing nodes are network peers at a layer of a distributed edge computing system.

6. 1. A method for cloudlet migration performed among a plurality of edge computing nodes in an edge computing environment, the method comprising: encrypting data related to the workload in a local data store within the source cloudlet to create encrypted data; transferring a key for the encrypted data from the source cloudlet to a destination cloudlet; receiving an indication of a workload migration; in response to the workload migration indication, transmitting the encrypted data to the destination cloudlet over a non-secure channel without re-encrypting the encrypted data for network transmission; method.

7. The method of claim 6 , wherein the encrypted data is transferred from the local data store to a working memory as encrypted data.

8. The method of claim 7 , wherein the source cloudlet decrypts the encrypted data using full-memory encryption circuitry as the data is transferred from the working memory to a processor.

9. The method of claim 6 , wherein a portion of the encrypted data is not transferred as part of the workload migration, the portion being determined by profiling data usage by the workload.

10. The method of claim 6 , wherein the edge computing node is a network peer at a layer of a distributed edge computing system.

11. At least one non-transitory machine-readable storage medium having instructions stored thereon that, when executed by a processor circuit of an edge computing system, cause the processor circuit to perform operations of the method of any one of claims 6 to 10.

Images