Distributed machine learning resource sharing and request routing

Abstract

The present application relates to distributed machine learning resource sharing and request routing. Various embodiments of the present disclosure improve existing multi-layer and other network technologies by routing and processing client requests that require machine learning based on the machine learning capabilities of each network device and / or other computer resource characteristics of different network devices. This ensures that network latency and throughput and other computer resource consumption characteristics will be improved as machine learning processing can occur at the most appropriate network device or distributed among various appropriate network devices.

Description

Technical Field

[0001] The present application generally relates to multi-tiered computer networks and more specifically to distributed machine learning resource sharing and request routing. Background Art

[0002] A multi-tiered computer network includes multiple layers of multiple networks or devices that interact with each other to service client requests. For example, a broadband access network may include an IoT layer of Internet of Things (IoT) devices (such as smart TVs or sensors), a customer premises equipment (CPE) layer of telecommunication hardware typically located in a customer's home or business, an edge layer of devices (such as mobile devices, micro servers, etc.) where the customer resides (or is closer to the customer or IoT device relative to the cloud) and controls and processes data at the boundary between two or more networks, and a cloud layer of computing nodes in data centers that perform most of the heavy processing and calculations. To service a client request, the request is typically routed to a device at each layer, and then the data is transmitted back to the IoT device.

[0003] Existing multi-layer computer networks and other computer networks do not provide functionality for sharing machine learning processing power and intelligently routing and processing client requests that require machine learning functionality among each other. In addition, existing technologies do not necessarily consume computing resources, such as memory, CPU, and network resources, among other things. Summary of the invention

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is neither intended to identify key features or essential features of the claimed subject matter nor to be used as an aid in determining the scope of the claimed subject matter.

[0005] Particular embodiments improve upon existing multi-layer and other network functionality by employing machine learning cores or modules across multiple network devices of a multi-layer network and sharing the machine learning capabilities (and / or other computer resource characteristics) of such modules across the network so that intelligent routing decisions can be made at runtime to service client requests that require machine learning processing. Client requests include machine learning data and control messaging. Thus, some embodiments route and process client requests that require machine learning based on the machine learning capabilities (e.g., model type, hyperparameters) of each network device and / or other computer resource characteristics (e.g., memory availability, CPU utilization, etc.) of different network devices. This ensures that network latency and throughput, as well as other computer resource consumption characteristics, will be improved as machine learning processing can occur at the most appropriate network device or be distributed among various appropriate network devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Aspects of the present disclosure are described in detail below with reference to the accompanying drawings, in which:

[0007] Figure 1 is a block diagram of a computing environment for implementing some embodiments of the present disclosure;

[0008] Figure 2 is a schematic diagram of an example operating environment architecture in which some embodiments of the present disclosure are employed;

[0009] Figure 3A is a block diagram illustrating an ML agent integrated into (or connected to) a cable modem (CM) according to some embodiments;

[0010] Figure 3B is a block diagram illustrating the integration or attachment of an ML agent to an RPD / RMD device according to some embodiments;

[0011] Figure 3C is a block diagram illustrating various ML agents resident in or connected to a CM gateway according to some embodiments;

[0012] Figure 4 is a block diagram of an example operating environment in which embodiments of the present disclosure may be utilized;

[0013] Figure 5 is a block diagram of an example network device illustrating how a machine learning core operates according to some embodiments;

[0014] Figure 6 is a block diagram of an example network device illustrating how a machine learning core operates to detect anomalies according to some embodiments;

[0015] Figure 7 is a block diagram of an example network device illustrating how a machine learning core operates to detect device-specific and / or network-wide PHY issues according to some embodiments;

[0016] Figure 8 is a block diagram of an example network device illustrating how a machine learning core operates to detect certain operational anomalies according to some embodiments;

[0017] Fig. 9 is a block diagram of an example network device illustrating how a machine learning core operates to detect and track a particular object according to some embodiments;

[0018] Fig.10 is a time series diagram illustrating that a video stream type may be detected and a sliding window may be used to determine the continuous presence of an object within a video stream according to some embodiments;

[0019] Fig.11is a schematic diagram illustrating how to use a long short-term memory (LSTM) model to process time series data (e.g., a video stream) according to some embodiments;

[0020] Fig.12 is a schematic diagram illustrating how one or more machine learning models may be used to generate decision statistics according to some embodiments;

[0021] Fig.13 is a block diagram of an example topology computing environment of an ML agent device and an ML client in a broadband access network infrastructure according to some embodiments;

[0022] Fig.14 is an example sequence diagram illustrating network devices that share ML resource information according to some embodiments;

[0023] Fig.15 is an example sequence diagram illustrating a network device establishing an ML service channel and handling a client request for processing data via one or more machine learning processes according to some embodiments;

[0024] Fig.16 is a block diagram illustrating how an ML request is routed from a client to a destination client device according to some embodiments;

[0025] Fig.17 is a block diagram illustrating an ML network (a) and an ML network (b) in a wired broadband access network infrastructure according to some embodiments;

[0026] Fig.18 is a block diagram of an example ML agent device according to some embodiments;

[0027] Fig.19 is a block diagram of an example client device according to some embodiments;

[0028] Fig. 20 is a flow chart of an example process for generating or updating a data object with data specifying computer resource characteristic information and machine learning capability information of a network device according to some embodiments;

[0029] Fig.21 is a flow chart of an example process for executing a client request to process data via one or more machine learning processes according to some embodiments;

[0030] Fig. 22 is a block diagram of a computing device according to some embodiments. DETAILED DESCRIPTION

[0031] The subject matter of aspects of the present disclosure is explicitly described herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this patent. Specifically, the inventors have anticipated that the claimed subject matter may also be embodied in other ways to include different steps or combinations of steps similar to the steps described in this archive in conjunction with other current or future technologies. In addition, although the terms "step" and / or "frame" may be used herein to refer to different elements of the adopted method, the terms should not be interpreted as implying any specific order between or among the various steps disclosed herein, unless and except for the order of the individual steps that are explicitly described. Each method described herein may include a computing process that can be performed using any combination of hardware, firmware and / or software. For example, each function may be implemented by a processor executing instructions stored in a memory. The method may also be embodied as a computer-usable instruction stored on a computer storage medium. The method may be provided by (to name a few) an independent application, a service or a managed service (independently or in combination with another managed service) or a plug-in for another product.

[0032] Existing multi-layer computer networks and other computer networks do not provide functionality for sharing machine learning processing capabilities among each other. Although some higher-level computing devices (such as cloud computing nodes) can perform machine learning processes, such processes are generally not fully distributed among different layers, such as at the edge or in CPE devices. Subsequently, processing throughput and network latency are adversely affected. For example, a client request may have to jump back and forth between multiple devices at each layer of the broadband access network before the request is finally processed at the cloud layer, which reduces the amount of bits per time segment of processed data in view of all the packet generation costs in protocols such as TCP / IP. In general, other transport protocols are also applicable.

[0033] Existing multi-layer networks and other computer networks also do not adequately route and process client requests that require machine learning processing. Although existing computing systems can route client requests using load balancing, fault handling, fault tolerance, or other routing functionality, this routing functionality does not take into account the machine learning capabilities of specific network devices.

[0034] Embodiments of the present disclosure provide one or more technical solutions to these problems with existing computer networks. In operation, various embodiments relate to the ability to employ machine learning cores or modules across multiple network devices of a multi-layer network and to share such modules across the network, so that intelligent routing decisions can be made at runtime to service client requests. For example, various gateways and edge devices in a broadband access network may employ different machine learning models with different hyperparameters and broadcast indications of the machine learning models and hyperparameters (e.g., via ML service advertisements) to other network devices, so that those other network devices can determine the machine learning capabilities or attributes of other network devices relative to their own machine learning capabilities and attributes.

[0035] Some embodiments of the present disclosure route and process client requests that require machine learning based on the machine learning capabilities of each network device and / or other computer resource characteristics of different network devices. For example, a client request may be to detect an object in a video stream. However, only certain network devices may employ an appropriate model (e.g., a CNN trained on images) to detect objects. And other network devices employ different models that perform different functionalities (e.g., a clustering model trained on malware signatures). In addition, a particular network device may be busy or otherwise experience a certain computer resource utilization or consumption that exceeds a threshold (e.g., memory availability, CPU utilization, etc.). Therefore, in this example, various embodiments of the present disclosure intelligently route client requests to those network devices with object detection capabilities and / or network devices that are not subjected to computer resource consumption that exceeds a certain threshold.

[0036] Various embodiments of the present disclosure improve existing multi-layer networks and other networks, which improves computer resource consumption. As described herein in conjunction with existing network technologies, machine learning functionality is not distributed among different layers, such as at the edge or in CPE devices. These technologies also do not fully route and process client requests that require machine learning processing. Therefore, processing volume and network latency are adversely affected. However, because multiple network devices include machine learning cores or modules, the ability information (such as machine learning model type), processing volume and latency shared across the network to route client requests are improved. This is because machine learning processing does not need to occur at a specified device or layer (such as a cloud layer). In addition, the computer resource characteristics (such as CPU, memory) of each network device can be determined to ensure that the request does not exceed a certain computer resource consumption threshold to be routed to those network devices. For example, the entire client request can be processed at the edge based on the machine learning capabilities of the corresponding edge network device to fully serve the request. In this example, because the client request does not need to jump to each layer and it is processed at the edge near where the client resides, network latency and processing volume are greatly improved. In another example, where heavy machine learning processing is required, for example, a first portion of a client request may be processed by a machine learning core at the CPE layer based on the machine learning capabilities of the CPE device, and then a second portion of the request may be processed at the cloud layer, such that different network devices can share the machine learning processing (e.g., in parallel), which also improves network latency and throughput.

[0037] Now go to Figure 1, a block diagram showing an example operating environment 100 in which some embodiments of the present disclosure may be adopted is provided. It should be understood that this and other arrangements described herein are stated only as examples. Other arrangements and elements (such as machines, interfaces, functions, sequences, and functional groupings) may be used as supplements or replacements for the arrangements and elements shown, and some elements may be omitted together for clarity. In addition, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or implemented in conjunction with other components and implemented in any suitable combination and implemented in any suitable location. Each function described herein as being performed by one or more entities may be implemented by hardware, firmware, and / or software. For example, some functions may be implemented by a processor executing instructions stored in a memory.

[0038] The operating environment 100 enables routing of requests 150 from client device(s) 102 to network device 112 for processing the requests via one or more machine learning functionalities according to some embodiments. The operating environment 100 includes one or more client devices 102, network device 112, network device 116, and one or more network devices 106 (representing any number of network devices), each of which is communicatively coupled via one or more networks 110.

[0039] Figure 1 The illustrative operating environment 100 shown in is only an example of one suitable operating environment and is not intended to imply any limitation on the scope of use or functionality of embodiments of the invention. For example, in some embodiments, computing environment 100 includes or represents a cloud computing environment, which includes a network-based distributed / data processing system that provides one or more cloud computing services (such as IAAS, PAAS, and SAAS). In addition, the cloud computing environment may include many computers (e.g., represented by network devices), hundreds or thousands of computers, or more, housed in one or more data centers and configured to share resources via (several) networks 110. In addition, the client device or the network device may be a module on the same device.

[0040] Additionally or alternatively, in some embodiments, operating environment 100 and one or more networks 110 represent or include any multi-layer and / or hierarchical computer network. "Hierarchical computer network" described herein refers to any computer network in which data is typically routed and / or processed from one network device to another in a hierarchical order, although the precise network device used to process the data is not necessarily predetermined. This example includes a broadband access network, which includes a home router, a set-top box, and a back-end server, wherein data is, for example, first routed from a set-top box network device, then routed to a home router network device, then routed to one or more back-end server network devices. Additionally or alternatively, a hierarchical computer network includes an Internet of Things (IoT) computing environment, wherein data is first processed by a CPE layer, then processed by an edge layer, and then processed by a cloud layer.

[0041] In various embodiments, the "layers" described herein refer to specific functions (e.g., edge computing) performed by a specific type or kind of network device (e.g., edge computing device, as opposed to a cloud computing node). In an IoT computing environment, for example, the layers may be an IoT device layer, a CPE layer, an edge layer, and a cloud layer. In some embodiments, the computer network(s) 110 represent or include one or more of: a local area network (LAN), a wide area network (WAN), the Internet, a telecommunications network, and / or the like. Figure 1 , network device 116 is included in layer 1 (e.g., CPE layer), and network device 112 and (several) network devices N 106 are included in one or more layers N (e.g., layer 2, such as an edge layer configured to process data after layer 1 processes the data). Layer N represents any different number of layers, such as 3 layers, 4 layers, or 5 layers. In some embodiments, network device 112 and one or more network devices 106 are included in the same layer, while in other embodiments, they are located in different layers.

[0042] The illustrative operating environment 100 includes one or more client devices 102. In some embodiments, the one or more client devices 102 represent any suitable device that generates and sends requests for processing data, and indicates processed data or execution functionality (e.g., notifications, messages, processed image data, etc.) of the requests back to the one or more client devices 102.

[0043] In some embodiments, client device(s) 102 represent any suitable computing device associated with a user, such as a smart television, a user's mobile phone device, a wearable sensor worn by a user (e.g., a pulse oximeter), and the like. For example, client device(s) 102 may represent a desktop computer, a portable computer, a laptop or notebook computer, a tablet computer, a pocket computer, a phone, a smartphone, a smart watch, smart glasses, or any other suitable type of electronic device associated with or worn by a user.

[0044] In some embodiments, one or more client devices 102 include or represent one or more network devices in addition to network devices 116, 112, and network device N. A "network device" as described herein refers to any suitable computing device configured to process, execute, and / or route a specific request originating at a client device. Examples of network devices include proxy devices, such as routers, set-top boxes, Internet access gateways, routers, network switches, home gateways, edge devices, cloud nodes, and the like. A "proxy device" as described herein refers to a gateway (e.g., a router) between a user device (e.g., a user's TV remote control, mobile phone, or client device 102) and a specific endpoint (e.g., a cloud computing node) network device configured as the final device for processing a request. For example, a proxy device may be an intermediary server that acts as a gateway between an end user and the website (or web server) they browse. Proxy devices that include machine learning cores or capabilities are referred to herein as "ML agents" or "ML proxy devices."

[0045] Other examples of network devices include a blade server, a web server, an application server, a mainframe, a cloud computing node, a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smart phone, or any programmable electronic device configured to process and / or route client device requests.

[0046] Operating environment 100 illustrates that one or more client devices 102 send a request 150, which may be a request to process data via one or more machine learning processes. For example, request 150 may be a request to process image data or a video stream sequence (e.g., a request to detect and follow a specific object). In some embodiments, request 150 is a request to determine any specific anomalies, such as malicious network intrusions or data privacy violations. In another example, request 150 may be a request to detect a specific physical layer ("PHY") problem (e.g., a channel condition metric such as SNR, MER, etc.).

[0047] Network device 116 receives the request and determines, via routing determiner 104, which network device to route request 150 to. Figure 11, route determiner 104 uses route resource list 120, which includes attributes for each network device in operating environment 100 (i.e., network device 116, network device 112, and network device(s) 106). In some embodiments, route resource list 120 (also referred to herein as "resource information") or a portion of route resource list 120 (e.g., sub-resource list 120-1) is shared between each of network devices 116, 112, and 106. Such attributes of route resource list 120 include one or more computer resource characteristics of each respective network device (e.g., CPU, memory, data transfer rate) and one or more machine learning processing capabilities of each respective network device (e.g., machine learning model type, hyperparameters used in the model). In some embodiments, some or each of network device 112 and network device(s) 106 additionally include the same route determiner as route determiner 105, so that they can also use their own version of route resource list 120 to make a routing determination for the request. It should be understood that routing resource list 120 is merely representative and may alternatively or additionally include any suitable attributes as described in greater detail below.

[0048] "Computer resource characteristics" refers to any specific state of any computing component in terms of hardware, software and / or signal processing of a corresponding network device. For example, computer resource characteristics may be any of the following: neural network type / architecture, data transmission rate of the network device, transmission protocol of the network device (e.g., User Datagram Protocol (UDP), Transmission Control Protocol (TCP), Reliable Data Protocol (RDP), Apple Telephone Transport Protocol (ATP), etc.), latency introduced by the network device and corresponding data path, internal and external memory capacity and bandwidth, processing capacity (e.g., Tera Operations Per Second (TOPS)), security level or type of input / output data, operational server memory in a cluster, number of servers (or nodes) in a cluster, storage availability within a cluster, actual memory utilization, CPU utilization, network utilization, and the like.

[0049] "Machine learning processing capabilities" (also referred to herein as "ML resources" or "ML resource information" or "ML capabilities") refer to any information that indicates specific machine learning functionality or other attributes associated with machine learning that a particular network device is capable of performing. As described herein, each network device includes a machine learning core that is capable of servicing and processing client requests. These ML capabilities may be shared or distributed across multiple network devices. For example, machine learning capabilities may include supported data paths for ML input and output data streams, supported ML decision statistics or inference types and / or corresponding hyperparameters (e.g., neural network class (e.g., CNN, LSTM, etc.), prediction targets (e.g., predict image features, predict malware, predict PHP issues, etc.), size of the neural network, neural network topology (e.g., number of layers or neurons, whether fully connected or feed-forward, learning rate)). Other examples include an indication of whether the model is a clustering model, a classification model, a regression model, supported ML input data pre-processing functions (e.g., normalization and scaling) and ML output data post-processing functions (e.g., return notifications of detected malware, return detected objects with bounding boxes), ML model provisioning and storage, and ML engine or ML core execution.

[0050] In some embodiments, based on what request 150 requires in terms of machine learning (e.g., object detection) and / or based on comparing the properties of different network devices (i.e., machine learning capabilities and / or computer resource characteristics), network device 116 routes request 150 to network device 112 so that network device 112 can process request 150 via its machine learning core 114 (or route request 150 to a network device that is more suitable for processing request 150). Figure 1 , each network device has its own machine learning core (i.e., 118, 114, and 108). The "machine learning core" described herein refers to a module that represents or uses one or more machine learning models that make a certain decision statistic or inference (e.g., classification prediction using a classification model, cluster prediction using a cluster model, or regression prediction).

[0051] In an illustrative example of routing of request 150, network device 116 may determine that request 150 requests processing of an object in a video stream and that its machine learning core 118 (or machine learning core 108) is unable to process the request based on the inclusion of different model types and / or hyperparameters required to process request 150. For example, machine learning cores 118 and / or 108 may be configured to process requests to detect PHY device specific issues, as opposed to image data. However, routing determiner 140 may determine via resource list 120 that network device 112 is capable of processing objects in video streams (or more specifically, ML core 114 includes CNN or other machine model functionality and the like configured to detect objects). Additionally or alternatively, actual network device computer resource characteristics, such as CPU utilization, memory capacity / availability, may be used. For example, network device 112 may currently have a small CPU utilization. In this way, routing of requests may be based on the machine learning or ML core capabilities and / or computer resource characteristics of each network device. Therefore, network device 116 may responsively forward request 150 to network device 112. In response, network device 112 processes the request via ML core 114 (eg, generates a bounding box over the object in the image and zooms in).

[0052] Network device 112 responsively sends processed data and / or notification 160 back to client device(s) 102 for execution of request 150. For example, network device 112 may cause a bounding box to be displayed on a moving person object as the person moves on a television.

[0053] As described herein, ML resources or "ML capabilities" may be defined as logical high-level resources responsible for supporting ML applications across networks. This may include: a) a list of specific IP data networks, referred to as ML resource networks; b) characteristics of each network node in the specified data network; and c) the (available) ML processing resources (i.e., machine learning capabilities) of the integrated ML core within each network node. In various embodiments, the ML resource network is rooted at the ML client where the ML application resides, and is dynamically formed according to the processing requirements of the ML application. In some embodiments, for a particular ML resource network, the resource list 120 and / or request 150 may be distributed and organized across multiple hierarchical (conceptual) network layers, such as CPE, edge, and cloud layers.

[0054] In some embodiments, machine learning capabilities include supported data paths (for ML input / output), ML inference types (e.g., neural network classes and corresponding hyperparameters), and for each data network node, data transfer rates, ML processing capacity, security capabilities, latency, and memory footprint and bandwidth. In some embodiments, machine learning capabilities (e.g., resource list 120) and / or requests 150 are distributed throughout a hierarchical multi-layered data network infrastructure and optimally shared among all ML clients residing within the infrastructure. In some embodiments, processing of requests 150 is achieved by dynamically routing ML requests from ML clients (e.g., client device(s) 102) to the most appropriate network device that includes the most appropriate network capabilities (via the corresponding ML core) that can implement the requested ML processing at the time of the request under specific network conditions (e.g., computer resource characteristics and / or machine learning capabilities). In some embodiments, such routing decisions are made by the ML resource hosting server (or simply the ML server) hop by hop along the data path from the ML client to the destination ML server according to these configurable routing policies of the server.

[0055] Various embodiments of the present disclosure utilize a concept referred to as an ML agent as an example of an ML server. Specific embodiments relate to effectively sharing ML capabilities (e.g., resource list 120) across, for example, a hierarchical, multi-layered data network infrastructure. In some embodiments, an ML agent is a device with a configurable ML core that performs multi-task ML processing on behalf of an IoT device that has its own limited ML resources. In some embodiments, the ML agent functionality is implemented in a home broadband gateway (e.g., a cable modem, a digital subscriber line (DSL) or a passive optical network (PON) gateway), a broadband access node (e.g., a DOCSIS remote PHY device / remote MACPHY device, a digital subscriber line access multiplexer (DSLAM), an optical line terminal (OLT)), or in any standalone dedicated device in a local area network or broadband access network.

[0056] Some embodiments utilize any IoT protocol, such as the MQ Telemetry Transport (MQTT) protocol, the Constrained Application Protocol (CoAP), the Hypertext Transfer Protocol version 3 (HTTP / 3), etc. As an example, various embodiments may utilize the MQTT protocol and the CoAP protocol that are co-deployed on IoT devices. Even though various embodiments of the present disclosure are described using a wired network, they may also be applicable to other infrastructures such as infrastructures for xDSL and xPON technologies. In addition, although various embodiments of the present disclosure target high-volume home ML applications, especially ML applications that can be enabled on home IoT devices, the functionality may also be applicable in enterprise environments or other client devices with similar characteristics.

[0057] In some embodiments, there are two schemes for processing client requests 150, sharing machine learning capabilities, and / or sharing computer resource characteristics (e.g., resource list 120). The first scheme may include distributed assignment of ML models. For a given ML application on an IoT device (e.g., client device(s) 102), a specific ML agent (e.g., network device 112) is dynamically assigned (e.g., at runtime) to support the entire ML processing session of the ML application. For example, a set of ML models may be distributed among ML agents (e.g., network devices 116, 112, and 106) across edge / gateway layers (e.g., layer 1 and layer N), and ML requests (e.g., including request 150) are routed from each ML application and routed to the agent or network device hosting its corresponding model (e.g., request 150 is only processed by the machine learning core 114 of network device 112 and no other network devices). The distribution of models may be done in such a way that the capabilities of the hosting agent are compatible with the hosted model.

[0058] In some embodiments, the second scheme is referred to as “distributed partitioning of ML models”. In these embodiments, multiple ML agents or network devices across a network operate together to achieve a given inference task or decision statistics by using local input data from a set of distributed IoT devices. In these embodiments, multiple machine learning models can be used to produce the final inference or decision statistics. For example, the classifier model hosted on the first network device may be a multimodal model (receiving heterogeneous inputs, such as image data and natural language characters) that classifies data based on predictions made by other models. This scheme is also referred to as “distributed inference”. In these embodiments, the input data is distributed among IoT devices (such as (several) client devices 102), and the ML processing associated with a given ML model (such as inference) is performed jointly by multiple ML agents. For example, request 150 may be processed by both the machine learning core 118 of network device 116 and the machine learning core 114 of network device 112. This scheme is useful for ML applications that require large-scale ML models and distributed input data across the network.

[0059] Figure 2 2 is a schematic diagram of an example operating environment architecture 200 in which some embodiments of the present disclosure are employed. In some embodiments, operating environment 100 represents operating environment architecture 200 or is included in operating environment 100. For example, network devices 203, 205, 207, and / or 210 represent Figure 1 One or more of the network devices.

[0060] like Figure 2 It is described that each machine learning core or process is hosted by each agent device and distributed across multiple layers in a hierarchical data network infrastructure. Figure 2 The illustration layers include a cloud layer, an edge layer, a CPE layer, and an IoT layer. The IoT layer includes IoT devices, such as a washing machine 220, a car 222, and a sensor 224 (e.g., a wearable sensor that measures oxygen saturation). It should be understood that these IoT devices are merely representative and any suitable IoT device may be used, such as a smart home lock system, a smart lighting system, a virtual assistant (e.g., a smart speaker), a smart doorbell, and / or the like. An "IoT device" or "IoE device" is any suitable article or machine that is programmed for a specific application (e.g., turning on a thermostat, washing clothes, turning on a light) and is equipped with a transmitter to transmit sampled or sensed data to another network device for processing of the sampled or sensed data.

[0061] The CPE (i.e., "customer premises equipment") layer of environment 200 includes CPE devices 203, 205, and 207, which correspond to telecommunications hardware typically located in a customer's home, business, or other location. For example, a CPE device may include a cable or satellite TV set-top box, a digital subscriber line (DSL) or other broadband Internet router, a VoIP base station, a telephone handset, or other customized hardware used by a particular telecommunications service provider.

[0062] The edge tier of environment 200 includes various edge nodes, including edge node 209. An "edge node" is a local processing computing device configured to store and process a subset or all of a received or corresponding data set over a network before routing the data to the cloud tier.

[0063] In some embodiments, the edge layer represents a mesh or other network of micro data centers or edge nodes that process and store local data received from the IoT layer and push or transmit some or all of the data to a cloud device or enterprise data center. In some embodiments, the edge node stores all data and transmits only selected (e.g., data that meets a threshold) or important data to the cloud layer. For example, unimportant data or data in a group that does not meet a threshold is not transmitted. For example, a lidar, radar, and / or camera sensor may sample map data but only push a portion of the map data. Therefore, only after a condition or threshold has been met, the edge node transmits data that meets or exceeds the threshold to the cloud node so that the cloud node can take responsive measures, such as notifying the client device to indicate that the threshold has been met.

[0064] In certain embodiments, data that does not meet or exceed a threshold is not transmitted to the cloud layer. In various embodiments where a threshold or condition is not met, daily or other period reports are periodically generated and transmitted from the edge node to the cloud layer, indicating all data readings collected and processed at the edge node. In some embodiments, the edge node acts as a buffer or gateway between its network and a wider network, such as located in the cloud layer. Thus, in these embodiments, the edge node may be associated with one or more gateway devices that translate proprietary communication protocols into other protocols, such as the Internet Protocol.

[0065] Figure 2 Network devices 203, 205, 207, 209, and 211 are illustrated as each having a machine learning core 206, 204, 202, 208, and 210, respectively. Thus, as described herein, requests can be routed to a particular network device based at least in part on the ML core capabilities. For example, a request originating from IoT device 220 can be routed to network device 203, and from network device 203 to edge node 209, and from edge node 209 to cloud node 211 based on the machine learning capabilities of network devices 203, 209, and / or 211.

[0066] Figure 2 A hierarchical data network infrastructure is described. Examples of such network infrastructure include broadband access networks. In these broadband access networks, the proxy device may be, for example, a home broadband gateway, a broadband access node, or a combination of both. Example broadband access systems include cable, xDSL, xPON, etc.

[0067] Figures 3A to 3C The following describes different locations where an ML agent device (e.g., a network device) can be placed. More specifically, since the ML agent device includes a machine learning core, Figures 3A to 3C In some embodiments, operating environment 100 includes or represents Figures 3A to 3C For example, client device(s) 102 may include ML client 302. In another example, each of network devices 116, 112, and 106 may be represented by network devices 306, 308, and 310.

[0068] Figure 3A 304 is a block diagram illustrating an ML agent integrated into (or connected to) a cable modem (CM) according to some embodiments. Specifically, for example, the ML agent 304 is integrated into or connected to a CM gateway device 306 (e.g., Figure 1 or in Figure 2Thus, the gateway device 306 serves ML clients (eg, stored to one or more client devices 102) that are attached to the CM gateway device 306 (eg, via a LAN).

[0069] like Figure 3A , the remote PHY device (RPD) / remote MAC device (RMD) 308 and the converged cable access platform (CCAP) device 310 do not include ML proxy functionality. The (several) devices 308 correspond to a method for a distributed access architecture (DAA). These devices 308 move components away from the CMTS head end. PHY moves the physical layer to the edge of the network, while MAC-PHY, for example, moves the media access control (MAC) layer to the edge. The PHY layer or physical layer defines the physical and electrical characteristics of the network. The physical layer is responsible for managing the hardware that modulates and demodulates the RF bits. The MAC layer is responsible for sending and receiving RF frames. Typically, as part of each data packet, there is a MAC layer data header with addressing information and data packet options. The MAC layer implements data packet acknowledgement (ACK), data packet tracking to eliminate duplication, and the like.

[0070] Figure 3B is an illustration of an ML agent 348 integrated or attached to an RPD / RMD device 350 (with Figure 3A ML agent 348 serves ML clients (eg, ML client 340) in client devices attached to a CM gateway (eg, CM gateway 346). Figure 3C 374 and 384 and RPD / RMD 382.

[0071] like Figure 3A , 3B 3C, ML processing resources (e.g., ML core and / or ML proxy devices) are distributed across various network devices. ML proxy can serve each ML client and can be shared by ML clients across CM gateway boundaries. In some embodiments, ML proxy devices can be integrated with data network CPE and node devices, and can be implemented as dedicated standalone devices connected to CPE or node devices and residing in a LAN or broadband access network. In some embodiments, Figure 3A , Figure 3B and Figure 3C Instruction requests may be routed to different ML agents or network devices based on the machine learning capabilities of the ML core within each ML agent and / or computer resource characteristics, such as for example Figure 1 describe.

[0072] Figure 4 is a block diagram of an example operating environment 400 in which embodiments of the present disclosure may be utilized. In some embodiments, Figure 1 The operating environment 100 includes or represents Figure 4 The operating environment is 400. Figure 4 In the operating environment 400, machine learning resource information (e.g., resource list 120) is distributed across three subnetworks, namely, a cloud ML agent subnetwork 403, an edge ML agent subnetwork (which includes subnetworks 405, 407, and 409), and a gateway ML agent subnetwork (which includes subnetworks 411 and 413).

[0073] The cloud proxy subnetwork 403 is a top-level subnetwork that includes a cloud ML proxy server 420 (e.g., network device 106) and a set of connected edge ML agents 424, 428, and 432 (e.g., network device 112). The edge ML agents 424, 428, and 432 can make their own ML resource information (e.g., sub-resource list 120-1) and / or computer resource characteristics available to the cloud ML proxy server 420. The cloud ML proxy server 420 makes its available ML resource information (e.g., sub-resource list 120-1)

[0074] 120-2) and / or computer resource characteristics are exposed and shared to all edge ML agents in the edge ML agent subnetwork

[0075] 426, 428 and 432. In some embodiments, this ML resource information includes the cloud ML proxy server 420 itself

[0076] Resource information and resource information exposed by edge ML agents 424, 428 and 432. Therefore, the following

[0077] ML resource information is physically available to each edge ML agent: (1) the edge agent’s own resource information (e.g., ML

[0078] 4), (2) resource information of the cloud ML proxy server 420 itself, (3) resource information exposed by the cloud proxy server 420 from other edge agents, and the like. In some embodiments, the cloud ML proxy server 420 is responsible for authenticating the edge ML agent (e.g., 424, 428, or 432) requesting any of the available ML resource information of the cloud ML proxy server 420. In this way, each network device within the cloud ML proxy subnetwork 403 may include its own ML client (i.e., clients 422, 426, 430, and 434) in order to request and obtain resource information from each network device, so that each network device has a complete list of each other's resource information (e.g., complete resource list 120). This allows each of the network devices 424, 428, 432, and 420 to be a candidate for routing requests (e.g., request 150) based on resource information.

[0079] Each of the edge proxy sub-networks (405, 407, and 409) includes edge ML proxy servers (i.e., 424, 428, and 432) and a set of connected gateway ML proxies 440 and 442. Each gateway ML proxy 440 and 442 in the sub-network makes its own resource information available to each respective edge ML proxy server 424, 428, and 432. Each edge ML proxy server 424, 428, and 432 exposes and shares its available ML resource information to all gateway ML proxies 440 and 442 in the sub-network. For example, this resource information may include available resources (e.g., memory consumption, CPU, etc.) of each edge ML proxy as well as resources aggregated from all gateway ML proxies 440 and 442. Thus, in some embodiments, the following ML resource information may be recursively available at each gateway ML proxy in sub-networks 405, 407, and 409: the gateway ML proxy's own resource information (e.g., sub-resource list 120-3), the gateway ML proxy's own resource information, the cloud proxy server 402's own resource information, those network devices exposed by the edge ML proxy server from other gateway ML proxy servers in the edge ML proxy's sub-network, and network devices exposed by the cloud proxy server 402 from other edge ML proxy servers. In some embodiments, each edge ML proxy server is responsible for authenticating the gateway ML proxy requesting any of its available ML resources.

[0080] The gateway proxy subnetwork (411 and 413) includes gateway ML proxy servers 440 and 442 and a set of connected IoT clients (e.g., clients 450 and 460). Gateway ML proxy servers 440 and 442 advertise and share their resource information to all IoT clients in the subnetwork. In some embodiments, the following resource information may be recursively available at the gateway ML proxy 440 and 442: the gateway ML proxy 440 and 442's own resource information, the gateway connected edge proxy server (e.g., 424)'s own resource information, the cloud ML proxy server 420's own resource information, the resource information exposed by the edge ML proxy (e.g., 424) from other gateway ML proxies (e.g., 440) in the subnetwork of edge proxies, and the resource information exposed by the cloud ML proxy server 420 from other edge ML proxies (e.g., 428). In this subnetwork and in some embodiments, the gateway ML proxy (e.g., 440) is responsible for authenticating IoT clients (e.g., 460) that request any of its available resource information.

[0081] like Figure 4 4, ML clients (e.g., 422, 426, 430, 434, 460, and 450) are hosted not only at IoT devices (e.g., client device(s) 102), but also at edge ML proxies 424, 428, and 432 and cloud ML proxy server 420. Each ML client may be responsible for requesting data to be processed (e.g., via machine learning functionality) and / or requesting resource information from another network device for routing.

[0082] Figure 5 is a block diagram of an example network device illustrating how a machine learning core operates according to some embodiments. In some embodiments, Figure 1 Network devices 116 , 112 , and / or 106 (and / or any network devices described herein) are represented as network device 502 .

[0083] like Figure 5 As illustrated in FIG. 5 , various embodiments of the present disclosure integrate a machine learning core 508 into a network processor 506 in an access network system-on-chip (SoC). ASoC is an integrated circuit that integrates all or most components of a computer (e.g., CPU, memory, I / O ports, secondary storage, etc.) on a single substrate or microchip. Integrating these various components into a single substrate or chip has a variety of advantages, such as lower power consumption and smaller geometry, which can be beneficial when these chips are integrated into smaller or lower-power devices such as set-top boxes, routers, and the like.

[0084] The machine learning core 508 is configured for ML processing within the access network infrastructure, and its capabilities or information can be shared with other network devices to enable ML clients distributed across the access network to receive data in response to their requests.

[0085] The processing modules used by the integrated ML core 508 (e.g., ML core 114) include a pre-processing module 514, a local inference module 512, and a post-processing module 510. A SoC data source 516 is fed into the pre-processing module 514 as input data. Examples of the SoC data source 516 include interfaces, functional modules, and software entities. In some embodiments, the data in the SoC data source 516 includes any data used to train a machine learning model, such as images for object detection, malware signatures for detecting malware, and the like.

[0086] The preprocessing module 514 is configured to reformat, normalize, and / or perform any other ML feature engineering functions. For example, the preprocessing module 514 may convert raw input data from the SoC data source 516 into a clean (e.g., usable) data set. In some examples, preprocessing may include or be followed by data collation (or data reprocessing). For example, the training component may use data collation during the training process of generating a trained model. Data collation may be used to transform and / or map raw training data into different formats that are more appropriate and / or more valuable for downstream purposes (e.g., analysis). For example, when the training component is training a model (e.g., after one or more iterations), it may be determined that a different format for one or more types of raw training data and / or training data is more suitable for downstream purposes than the format currently being used. By reformatting this data, the performance of the trained model may be more effective. In the example illustration, natural language words may become real numbers (e.g., 0 or 1) to be represented as feature values ​​in a vector space.

[0087] The local inference module 512 is generally responsible for generating decision statistics or inferences. For example, if a classification model is being used, the local inference is a classification prediction (e.g., object X is detected as category Y). If a clustering model (e.g., K-means) is being used, the local inference is a clustering prediction. If a regression model is being used, the local inference is a regression prediction. In some embodiments, the inferences performed by the ML model are provisioned in the non-volatile memory of the access network SoC 504 or dynamically downloaded at runtime.

[0088] The post-processing module 510 is generally responsible for reformatting the ML inference output data. For example, post-processing may include data anonymization (e.g., obfuscation, encryption, masking, etc.) of usernames or other sensitive data (e.g., social security numbers, addresses, etc.).

[0089] In various embodiments, the processing results of the ML core 508 are used to directly trigger interactions with local user devices, such as one or more client devices 102 (e.g., via voice utterances on a virtual assistant, video on a television, gestures, graphics, remote control notifications, etc.). For example, in response to the post-processing module 510 performing its functionality, the network device 502 may alert the local user device of a detected event (e.g., an object or detected malware) and suggest actions that the local user can take (e.g., zooming in on the object). In some embodiments, the ML core 508 (or an associated network device 502) generates a notification based on the local processing results of the ML core 508 and transmits the notification to a cloud server (e.g., Figure 2 211) for further analysis. In some embodiments, this analysis can be used for a specific access network device. In some embodiments, notifications from multiple access network devices can also be used for joint analysis (e.g., for correlation analysis). Correlation analysis can analyze the strength of the relationship between two or more numerically measured and continuous variables (e.g., height and weight).

[0090] The ML core 508 can be used in various ML applications. For example, applications can be developed using the integrated ML core to help network operators diagnose network-related problems to improve operational efficiency and / or detect audio / service anomalies to improve customer satisfaction. In another example, ML resource information available within the access network infrastructure can be used to support various ML applications running on CPE devices within a home or business, as described herein. Figures 6 to 10 Describes various applications or machine learning functionality that may be performed by a machine learning core according to various embodiments.

[0091] Figure 6 is a block diagram of an example network device illustrating how a machine learning core operates to detect anomalies according to some embodiments. In some embodiments, Figure 5 The network device 502 includes information about Figure 6 Describes the components and functionality.

[0092] like Figure 6 6, ML core 608 processes IP stack network data 616 to diagnose one or more anomalies and / or activities of the access network via ML processing requests (e.g., request 150). In these embodiments, ML core 608 and IP stack 615 (network data parameters) are co-located in the same access network SoC 604. Specifically, IP network data parameters within IP stack 616 are passed to pre-processing module 614, then to local inference module 614, and then to post-processing module 610 to generate notification or send data.

[0093] An "anomaly" may be a deviation within the data (e.g., IP stack 616) that exceeds a certain threshold. Examples include connectivity, media content leakage, malicious network intrusion, and the like exceeding a certain threshold. Examples of local inferences made by the local inference module 612 may be the following. Diagnosis of connectivity issues that make up a sub-network (e.g., IoT network, home network, edge network, etc.). Connectivity issues may include, for example, unexpected network failures at a network device (e.g., due to failures at one of the SYN, ACK, SYN-ACK steps for handshaking), duplicate IP addresses, IP address exhaustion, DNS issues, a single workstation that cannot connect, etc. Other examples of local inferences made by the local inference module 612 may be detection of media content leakage (e.g., media being broadcast to unauthorized devices), data privacy violations (e.g., names, social security numbers, or other private information being leaked), detection of malicious network intrusions (e.g., detection of malware such as injection attacks), identification of source and destination websites for audio / video media content transmitted across access networks.

[0094] Figure 7 is a block diagram of an example network device illustrating how a machine learning core operates to detect device-specific and / or network-wide PHY issues according to some embodiments. In some embodiments, Figure 5 The network device 502 includes information about Figure 7 The PHY data parameters at the PHY interface 716 are passed to the ML core 708, and the ML core 908 processes the ML processing request (via the pre-processing module 714, the local inference module 712, and the post-processing module 712).

[0095] PHY is short for physical layer or medium. It is usually layer-1 in the OSI stack. It interfaces the physical medium with the MAC and upper layers. The physical medium can be copper wire, fiber optic cable, twisted pair and / or wireless channel. More specifically, PHY is a circuit system unit that connects the link layer device (such as MAC) to the physical medium, as described above. PHY devices usually include a physical coding sublayer (PCS) and a physical medium dependent (PMD) layer. The PCS encodes and decodes the data being transmitted and received. Therefore, local inferences or decision statistics of the PHY or physical medium can be made from the machine learning model based on the characteristics of each physical medium, PHY and / or MAC.

[0096] Examples of local inferences made by the local inference module 712 may be the following, detection and classification of network impairments (e.g., via a classification-based machine learning model), such as the following classifications: signal constellation (e.g., based on symbols), RF spectrum (e.g., classification of specific RF values), channel noise (e.g., classification of specific noise values ​​or categories, such as specific frequency range levels), classified channel condition metrics (e.g., SNR, MER, etc.). Other examples include estimation and recommendation of network operating parameters (e.g., SNR margin, OFDM bit loading profile). Other examples include characterization of sub-network conditions (e.g., WiFi RF environment, home cabling conditions, etc.). In some embodiments, network impairment predictions include predictions of the timing or presence of: latency, jitter, packet loss, and the like. Examples of device-specific problems include predictions of: physical media (e.g., cable) being damaged based on "shooting stars" or flickers in video, fuzzy / blurry pictures (e.g., display resolution exceeds a threshold), no or intermittent pictures in video (e.g., exceeds a threshold), no or intermittent sound (e.g., exceeds a threshold), colors outside of normal color thresholds, and the like.

[0097] Figure 8 is a block diagram of an example network device illustrating how a machine learning core operates to detect specific operational anomalies according to some embodiments. In some embodiments, Figure 5 The network device 502 includes information about Figure 8 Specifically, the operating state data parameters of the SoC functional module 816 are passed to the pre-processing module 814, then to the local inference module 812, and then to the post-processing module 210 to generate notification or transmit data.

[0098] like Figure 8 8, ML core 808 processes SOC function module 816 to diagnose or detect one or more anomalies at SoC via ML processing request (e.g., request 150). Specifically, the operating state data parameters of SoC function module 816 are passed to pre-processing module 814, then to local inference module 814, and then to post-processing module 810 to generate notification or send data based on the ML processing request. In this way, anomalies on a chip (e.g., access network SoC 806) (e.g., on a user device) can be detected.

[0099] Each device accesses or stores important sensitive, personal data, including our bank and credit card information, email contacts, browsing history, location, and even private physiological information such as heart rate and sleep patterns. In addition to personalized end-user information, these devices contain highly confidential collateral from architecture, design, and manufacturing, such as encryption and digital rights management (DRM) keys, programmable fuses, on-chip debugging tools, de-signing bits, etc. Malicious or unauthorized access to secure assets in computing devices can lead to identity theft, leakage of corporate trade secrets, and the like. Therefore, the embodiments described herein can predict that specific anomalies will occur on the chip (i.e., on the access network SoC 806).

[0100] Examples include detection and prediction of specific security violations, such as predicting a specific malware signature stored to a device (e.g., based on historical malware signatures). In some embodiments, a recommendation for thermal throttling measures for the SoC following the prediction may be made. Typically, thermal throttling includes limiting the maximum frequency at which the SoC can operate. Throttling helps ensure that the SoC does not reach high temperatures. When the temperature rises above a threshold, the system may throttle performance, which reduces heat. With respect to some malware (e.g., viruses), for example, the malicious results cause the computer to heat up. However, as described herein, some embodiments may predict that specific malware is stored to a device based on learned malware signature features and responsively throttle the system.

[0101] Some embodiments additionally perform traffic load balancing or routing based on predicting specific anomalies at the SoC level. In this way, requests can be routed to network devices different from the anomaly detected at the access network SoC 806 so that malicious data will not be injected into the request and / or reduce heat or otherwise share the request load via load balancing.

[0102] Fig. 9 is a block diagram of an example network device that illustrates how a machine learning core operates to detect and track a particular object according to some embodiments. In some embodiments, Figure 5 The network device 502 includes information about Fig. 9 Describes the components and functionality. Fig. 9 The ML core 908 is illustrated as being applied to enable an end-to-end ML application with a pipeline of three (or more) distinct stages of inference operations, namely, via an object detection module 918, an object tracking module 912, and an object enlargement module 910. An "object" as described herein refers to a logical concept representing any logical entity having a unique signature (e.g., a set of features) embedded in the ML input data. For example, an object may refer to a unique set of pixels representing a car, a building, or another object in image data.

[0103] Object detection is the process of detecting and classifying a specific object. Specific embodiments use one or more machine learning models (e.g., convolutional neural networks (CNNs)) to generate bounding boxes that define boundaries and encompass computer objects (e.g., buildings, landscapes, bushes, etc.) that represent a specific location or features of the location. The bounding box describes or defines the boundaries of the object in terms of the location (e.g., 2-D or 3-D coordinates) of the bounding box (as well as the height and width of the bounding box). For example, a bounding box may be a rectangular box determined by its x- and y-axis coordinates. This gives the object recognition system an indicator of spatial distinction between objects to help detect the object. In some embodiments, one or more machine learning models may be used and trained to generate tighter bounding boxes for each object. In this way, the shape of the bounding box can change, and the confidence level of the classification / prediction can increase based on increased training sessions. For example, the output of the convolutional neural network (CNN) or any other machine learning model described herein may be one or more bounding boxes above each location feature, where each bounding box includes a classification prediction (e.g., this object is a building) and a confidence level (e.g., 90% probability).

[0104] Thus, the object detection module 918 detects and classifies the object(s) by detecting signature(s) associated with the objects present in the ML input data (e.g., SoC data source 916 and runtime ML processing requests.) In various embodiments, detection and classification includes determination of the presence of the object as well as any attributes associated with the object (e.g., its geometric location and shape within the input image, signal quality indicators within the PHY layer input data, etc.).

[0105] The object tracking module 912 is generally responsible for tracking objects in response to object detection via the object detection module 918. In various embodiments, the object tracking module continuously or periodically tracks the object across the input data samples. In this manner, a bounding box may be continuously placed over the object, for example, within each window or time series. In some embodiments, object tracking includes determining updated properties of the object (e.g., the new position and shape of the object in the input image stream, an updated signal indicator of the input PHY data, samples, etc.).

[0106] The object magnification module 910 is generally responsible for magnifying detected objects by scaling associated attributes (e.g., by magnifying pixels of a tracked image or images) and / or by detecting more detailed attributes (e.g., additional PHY layer signal quality indicators and metrics or additional sub-image objects of a larger object).

[0107] In some embodiments, post-processing module 910 merges the results from modules 918 and 912 and generates a final ML output result and / or notification, applying any necessary data privacy rules. For example, network device 902 may cause image data to be streamed to a client device when an object is moving (with a bounding box over the object in the image data) and with the object zoomed in relative to other image data outside the object. For example, a user may desire to get a close-up of a particular runner in order to see the runner's biomechanical movement. Thus, the client device may receive a request to zoom in on a particular runner as described and thus forward the request to a network device having particular machine learning capabilities and / or computer resources available to process the request, such as, for example, with respect to Figure 1 describe.

[0108] Fig.10 is a time series diagram illustrating that according to some embodiments, a video stream type may be detected and a sliding window may be used to determine the continuous presence of an object within the video stream. Fig.10 Description by ML core (e.g. Fig. 9 An example of an end-to-end ML process for classifying or otherwise predicting the type of a video stream (e.g., movie, sports, news) by using time series packet counts (e.g., the amount of TCP / UDP packets received / transmitted across a gateway device such as a cable modem) as ML input data, such as the ML core 908 of Fig.10 A "time series" is a sequence of data of any duration or interval (e.g., X-second time increments).

[0109] Fig.10 The time series packet counts are sampled for a given set of video stream types and used to train an ML model that classifies or predicts any sampled video stream into one of the video stream types in the training dataset, namely, video stream type 1 (e.g., sports), video stream type 2 (e.g., movies), video stream type 3 (e.g., cartoons), and video stream type 4 (e.g., news). In some embodiments, an object representing a video stream type is deemed to be "detected" if the classification or prediction probability for the video stream type exceeds a certain threshold (e.g., 60%). In some embodiments, in response to a video stream type being detected, it is tracked by running the ML model on the time series input data within a sliding window 1003, such as Fig.10In some embodiments, sliding window 1003 has the same duration as the duration of the time series input data. For example, time series 0 may have a duration of 5 minutes. Likewise, sliding window 1003 may also have a duration of 5 minutes. Sliding window 1003 thus allows tracking of objects within a video stream in a particular time series to determine that the object is continuously present in the video stream. In some embodiments, the tracked object may be magnified (e.g., via Fig. 9 910). For example, in a hurdle sporting event, a particular runner may be detected with a bounding box over the runner. Certain embodiments may magnify the pixels inside the bounding box or "zoom in" on the runner as the runner is performing a hurdle event across a time series.

[0110] In some embodiments, additional ML or random processing includes indicating or classifying a sub-type of a particular sampled video stream. For example, a movie genre may have sub-types such as "Cartoon," "Old West" movie, "Action" movie, "Comedy," "Romance," "Drama," and the like. Similarly, a type of sporting event may have sub-types such as "Football," "Soccer," "Tennis," "Hockey," "Basketball," "Baseball," and the like. In some embodiments, this end-to-end ML processing is enabled by the integrated ML core's directed access to the required data within the SoC.

[0111] Fig.11 1 is a schematic diagram illustrating how time series data (e.g., video stream) is processed using a long short-term memory (LSTM) model 1100 according to some embodiments. In some embodiments, the LSTM 1100 represents a Fig. 9 Machine Learning Core 908 and / or about Fig.10 functionality of the implementation. Fig.11 LSTM is described, but it should be understood that this is merely illustrative and there may be any number or type of suitable components. For example, in some embodiments, other models (such as CNN or Hidden Markov Model (HMM)) may be used instead of LSTM.

[0112] Fig.11Illustrates how feature vectors (e.g., representing a video stream) can be aggregated and concatenated sequentially. The LSTM model function assumes that downstream events (or later events, such as the last cell feature of a row) depend on and vary according to one or more events from the past (e.g., the previous image of the same video stream). In other words, the LSTM is a model that shares weights over time. In the context of table encoding, the model can learn cell feature patterns over time based on past cell feature observations (e.g., determine that the first three time series indicate a specific object, so the next cell in the same row may also be a specific object).

[0113] Element or unit 1101 represents a feature vector or other element representing a time series of values ​​for a particular data set. In an illustrative example, unit 1101 may represent a particular time series of a particular video stream, such as Fig.10 Unit state 1103 corresponds to a selective memory of past unit states (i.e., it takes data (e.g., vector values) that have been forgotten or ignored and data that have been remembered or saved) and passes the remembered data to the next unit or step (e.g., the next part of the video stream in the next time sequence). When the data reaches the last unit (e.g., representing the last part of the video stream in the last time sequence), C t With h t+1 Linear combination, so that h t+1 is the final output.

[0114] In various embodiments, the first operation occurs when the forget gate layer 1105 (S-type layer) determines what vector value to remove from the cell state 1103, as determined by f t =σ(W f ·[h t-1 ,x t ]+b f ) indicates that h t-1 (e.g., the previous unit state and the portion of the video stream corresponding to the previous feature value in the previous time series) and x t (current vector value) and outputs a number between 0 and 1. Values ​​close to or within a threshold of 1 indicate true or keep this data, while values ​​0 or within a threshold of 0 indicate forgetting or removing the data. The previous cell state indicates at least a portion of the vector value passed from the previous cell, as opposed to the overall cell state which indicates at least a portion of the data passed for at least a portion of the cell up to a particular point.

[0115] In various embodiments, the next operation will determine what information will be stored or moved along the cell state 1103, as determined by i t =σ(W i ·[h t-1 ,x t ]+b i ); This happens via an input gate layer 1107 (another S-type layer) and a hyperbolic tangent layer 1109. The gate layer 1107 determines which vector values ​​will be updated and the hyperbolic tangent layer 1109 generates new candidates C t In an example illustration, embodiments may forget table entry feature values ​​or vector representations that are: repetitive, noisy, below and / or above a certain threshold (e.g., exceeding a predefined number of words or other character sequences), and / or contain unparsed or encoded extraneous information, such as certain symbols and the like.

[0116] Next, certain embodiments convert the old cell state 1103 (C t-1 ) is updated to the new unit state C t , as by Therefore, the old cell state (C t-1 ) is multiplied by 1105, thereby forgetting the vector value described above. Next, an input gate layer 1107*hyperbolic tangent layer 1109 is added. In the final step, the content of the output is determined, which is determined by o t =σ(W o [h t-1 ,x t ]+b o );h t =σ t *tanh(C t ) represents the output based on the new cell state C t , which is a filtered version of the cell state. The S-type layer determines the cell state 1103 (C t-1 ) will be output. The cell state 1103 is run through the hyperbolic tangent to make the value between -1 and 1, and the final output h t+1 And this is multiplied by S-type gate 1111 so that only necessary values ​​are output. In various embodiments, the final output h t+1 A concatenated feature vector or context vector comprising some or each of the video stream feature values ​​aggregating one or more time series, as described herein. For example, the output may be a concatenated feature vector with a specific forgetting value for each time series representing a specific video stream.

[0117] As use Fig.11 As an alternative (or in addition) to the LSTM described in , some embodiments utilize other models such as hidden Markov models (HMMs) and / or Gaussian mixture models (GMMs) to process time-based data, such as video streams.

[0118] Fig.12 is a schematic diagram illustrating how one or more machine learning models may be used to generate decision statistics according to some embodiments. In some embodiments, Fig.12Represents any machine learning core described in this paper (e.g. Figure 5 The data and functionality of the machine learning core 508). Fig.12 It is illustrated that one or more of runtime input(s) 1203 are fed or processed by machine learning model(s) 1205 to generate decision statistics based on machine learning model(s) 1205 that have been previously trained on training input(s) 1215. Although runtime input(s) 1203 indicate specific inputs, they are merely representative and more or fewer inputs may be used.

[0119] Given (several) runtime inputs 1203, one or more machine learning models 1205 generate one or more specific decision statistical predictions 1207 (e.g., classification predictions for a classifier model, clustering predictions for a clustering model, or regression predictions for a regression model). The machine learning model(s) 1205 may be any suitable model of any suitable type. For example, the machine learning model(s) may be supervised or unsupervised and may be or include a neural network (e.g., a convolutional neural network (CNN) or a siamese neural network), a decision tree, a random forest, a support vector machine, a naive Bayes ( Bayes) and / or clustering (e.g., K-means clustering). Thus, while the machine learning model(s) 1205 are represented as neural networks, it should be understood that any suitable machine learning model (or combination of models) may alternatively or additionally be used. In the illustrative example of decision statistics(s) 1207, the machine learning model(s) 505 may cluster or classify feature vectors representing the video streams of the input(s) 1203 into clusters or groups representing particular video stream types (e.g., "motion" streams).

[0120] In certain embodiments, the decision statistics 1207 may be hard (e.g., membership in a class is a binary "yes" or "no") or soft (e.g., there is a probability or likelihood attached to a label). Alternatively or in addition, transfer learning may occur. Transfer learning is the concept of reusing a pre-trained model for a new, related problem.

[0121] In some embodiments, the machine learning model(s) 1205 converts or encodes the runtime input(s) 1203 and / or the training data input(s) 1215 into corresponding feature vectors in a feature space. A "feature vector" (also referred to as a "vector") described herein includes one or more real numbers, such as a series of floating point numbers or integers (e.g., [0, 1, 0, 0]) representing one or more other real numbers, natural language (e.g., English) words, and / or other character sequences (e.g., symbols (e.g., @, !, #), phrases, and / or sentences, etc.). This natural language word and / or character sequence corresponds to a feature set and is encoded or converted into a corresponding feature vector so that a computer can process the corresponding extracted features. For example, an embodiment may parse, tokenize, and encode each natural language word within the runtime input(s) 503 into a single feature vector.

[0122] In various embodiments, the machine learning model(s) 1205 learn parameters or weights via training so that similar features are closer to each other in feature space (e.g., via Euclidean or cosine distance). In some embodiments, this training is done in a supervised manner using a loss function (e.g., triplet loss or GE2E loss) that attempts to map similar features into a classification or cluster. Training can occur on any suitable training data input(s) 1215, such as labeled video stream types (e.g., such as Fig.10 ), tagged image types (e.g. Fig. 9 ), marked IP stack attributes (e.g., Figure 6 ), tagged PHY data (e.g., Figure 7 ) and / or tagged SoC data (e.g., Figure 8 Some or each of these inputs indicates whether the corresponding data indicates a particular classification or clustering based on features of the input (e.g., image features, malware signatures, etc.).

[0123] Various embodiments may represent one or more feature vectors representing input(s) 1215 in a vector space by aggregating (e.g., average / median or dot product) particular vector values ​​to arrive at a particular point in the feature space. For example, in a supervised learning context, a training component may receive a particular malware signature and a particular tag indicating that features of the training data input are indicative of a particular rootkit malware, such as "Rootkit X version" and the like.

[0124] Then, an embodiment may run the training input 1203 with corresponding labels through (several) machine learning models 1205, so that different feature values ​​and weights are learned according to the labels. In this way, when the training input 1215 is received, the corresponding weights or features may be learned. In some embodiments, the training component of the (several) machine learning model 1205 learns features from (several) training data inputs 1215 and responsively weights them during training. "Weight" in the context of machine learning represents the importance or significance of a feature or feature value used for prediction. For example, each feature may be associated with an integer or other real number, where the higher the real number, the more significant the feature used for its prediction. In some embodiments, weights in neural networks or other machine learning applications may represent the strength of the connection between nodes or neurons from one layer (input) to the next layer (input). A weight of 0 may mean that the input will not change the output, while a weight above 0 changes the output. The higher the value of the input or the closer the value is to 1, the more the output will change or increase. Similarly, there may be negative weights. Negative weights proportionally reduce the value of the output. For example, the more the value of the input increases, the more the value of the output decreases. Negative weights may contribute to negative scores.

[0125] In another illustrative example of a training component, some embodiments learn embeddings of feature vectors based on learning (e.g., deep learning) to detect similar features between (several) training data inputs 1215 in a feature space using a distance metric such as cosine (or Euclidean) distance. For example, each labeled training data input 1215 is converted from a string or other form into a vector (e.g., a set of real numbers), where each value or each set of values ​​represents an individual feature of a search result candidate or query in a feature space. A feature space (or vector space) is a collection of feature vectors that are each oriented or embedded in a space based on the aggregated similarity of the features of the feature vectors. During each training phase or period, specific feature characteristics of each (several) inputs 1215 may be learned or weighted. For example, for a first labeled malware signature, the most prominent feature may be a first behavior (e.g., a specific URL), while other features are substantially changed or absent, such as causing a pop-up ad to appear. Thus, patterns of the first behavior may be weighted (e.g., node connections strengthened to values ​​close to 1) indicating a signature that exhibits this feature (while other node connections representing the second set of symptoms are weakened to values ​​close to 0). In this way, embodiments learn weights corresponding to different features so that similar features found in the input contribute positively to the prediction.

[0126] In various embodiments, after the machine learning model(s) 1205 are trained on the training data input(s) 1215 (and / or tested), the machine learning model(s) 1205 (e.g., in a deployed state) receives the runtime input(s) 1203 (e.g., received from an IoT device). In various embodiments, the input(s) 1203 are automatically converted into one or more feature vectors and mapped in the same feature space as the vector(s) representing the training data input(s) 1215. In response, some embodiments determine the distance (e.g., Euclidean distance) between the one or more feature vectors and other vectors representing the training data input(s) 1215, which is used to generate the decision statistics(s) 1207.

[0127] In the illustrative example of decision statistics 1207, the feature vector of pixel values ​​representing the image of runtime input(s) 1203 may be closest to the feature vector representing the classification or cluster of "person." Thus, in this example, decision statistics 1207 may be a prediction that the object associated with runtime input(s) 1203 is classified as a "person" (e.g., based on comparing the features of the runtime image to the training images).

[0128] Fig.13 1 is a block diagram of an example topology computing environment 1300 of an ML agent device and an ML client in a broadband access network infrastructure according to some embodiments. In some embodiments, Figure 1 The computing environment 100 of FIG. 1 represents the environment 1300. In some embodiments, the computing environment 1300 represents any suitable environment described herein, such as Figure 2 The computing environment 200. Broadband access networks provide high-speed Internet access via various types of technologies such as fiber optic wireless, cable, DSL, and satellite. The minimum upload speed is usually a minimum of 25Mbps download and 3Mbps.

[0129] The environment 1300 includes an IoT device 1302, gateway devices 1304, 1312, edge devices 1306, 1310, and a cloud server 1308, each of which is communicatively coupled via networks 1322, 1338, 1340, and 1342. In some embodiments, the ML agent devices (i.e., 1306, 1308, 1310) first advertise or share their available ML resources (i.e., machine learning capabilities and computer resource characteristics). Subsequently, the ML client device (e.g., 1302) communicates with the ML agent device (e.g., ML agent device 1306) to which it is connected to establish an ML session between the ML application and the selected ML resource. Subsequently, the ML client device and the ML agent device transmit ML application data on the ML application data path between the ML application on the ML client device and the ML resource on the selected agent.

[0130] like Fig.13 As shown in , when ML application data is transmitted along the data path, it is processed by various components. For example, IoT device 1302 sends a request to process data via ML application 1314. This request is processed and encoded via IoT protocol stack 1316, local area network stack 1318, and then, through the lower level MAC / PHY layer 1320. The IoT protocol stack is considered to be an extension of the TCP / IP layered protocol model. The IoT protocol stack includes a physical layer, a link layer, a network layer, a transport layer, an application protocol layer, and an application service layer. In response to the request being processed by the MAC / PHY layer 1320, it is routed to the MAC / PHY layer 1334 via the home local area network 1322 (for example, a computer connected via Ethernet), and then, the MAC / PHY layer 1334 decodes the message and passes the message to the local area network protocol stack 1330, and the local area network protocol stack 1330 further decodes the message and passes it to the IoT protocol stack 1328, and the IoT protocol stack 1328 converts the message into a format that can be processed by the ML service management component 1326. As shown Fig.13 As described in , each network device has similar components to extract, encode and decode the corresponding request.

[0131] Each network device includes an ML service management component (e.g., 1326) that is responsible for managing ML resource information (e.g., 1304) and routing specific client requests. For example, before a client request is made via the ML application 1314, the gateway device 1306 may share its ML capability information and computer resource information to devices 1324, 1308, and 1310 via the ML service management component 1326. Each of the other devices may likewise share its ML capability information and computer resource information to the other devices via its respective ML service management component 1326. Subsequently, for example, the ML application 1314 may issue a request to process data via one or more machine learning processes. When this request is routed to the network device 1306, the ML service management component 1326 may look at its own ML resources 1304 as well as other ML resources of other network devices in order to make a determination as to which network device to route the request to.

[0132] Fig.14 is an example sequence diagram illustrating network devices that share ML resource information according to some embodiments. Fig.14In step 1, each network device (i.e., 1402, 1404, and 1406) advertises its ML service, i.e., ML service discovery advertisement 1408 (e.g., machine learning capability information and / or other computer resource information) to other network devices to which it is connected (e.g., communicatively coupled). This strict communication of connected network devices takes into account that in some embodiments, the ML service only allows child devices to be updated only from the parent device to which they are connected. The broadband network is, for example, a strictly (physical) hierarchical network, where "child" devices can only communicate directly with their "parent" device (e.g., IoT device to GW, GW to edge device, and edge device to cloud server). In these embodiments, due to the hierarchical architecture and for scalability reasons, the child devices are not aware of the overall network of ML resources (e.g., routing resource list 120) but only the ML resources advertised by their parent device. In various examples, an IoT device will always send a ML resource request to its own GW, which can either process the request with its own local resources (e.g., ML core) or propagate the request to its parent device (e.g., edge can then do the same thing but this will be hidden from the child devices). Thus, each network device along the chain makes its own "routing" determination to establish a virtual service channel between the ML application and the ML model.

[0133] exist Fig.14 In step 2, in response to receiving the ML service advertisement message from step 1, the corresponding network devices (i.e., 1404 and 1406) propagate the message (ML service advertisement) to the corresponding devices. Continuing with step 2, in response to receiving the ML service advertisement message from the cloud server 1406, the edge device(s) 1404 propagate the message to the GW device 1402 to which it is connected. In step 3, in response to receiving the ML service advertisement message, the corresponding network devices (i.e., 1404 and 1406) store the advertised ML service information in their corresponding ML service databases. In step 4, in response to receiving the ML service advertisement message, the ML agent 1402 (and all GW devices) store the advertised ML information in their ML service databases.

[0134] Fig.15 is an example sequence diagram illustrating a network device establishing an ML service channel and processing a client request for processing data via one or more machine learning processes according to some embodiments. In some embodiments, Fig.15 The step occurs in Fig.14 After the step of , and the network device is the same network device. For example, ML agent 1402 may represent ML agent 1504, ML agent 1404 may represent ML agent 1506, and ML agent 1406 may represent ML agent 1508.

[0135] In the ML service discovery advertising layer (e.g. Fig.14) After performing its functionality, the ML service channel establishment layer 1530 is executed to establish a communication session between the client device and one or more network devices and make routing decisions for the request. In various embodiments, the communication channel is an end-to-end virtual connection between the ML application client device and the device executing the ML model of the specific application. Thus, after the channel is established, the network devices between the two ends act as a bridge. Specifically, according to step 5 (which occurs at Fig.14 4 in step 6), the ML application on the IoT device(s) 1502 requests ML services to the gateway agent(s) 1504 to which it is communicatively coupled. According to step 6, in response to receiving the ML service request from the IoT device(s) 1502, the GW device(s) 1504 queries its ML service database to retrieve a list of ML agents (network devices) that support this service (e.g., object detection) and selects the most suitable one. According to step 7, if the GW device at step 6 is not the selected ML agent, the ML agent establishes a logical channel with the selected ML agent through the edge device and cloud server routing network.

[0136] According to step 8, in response to the reception of the logical channel establishment message, if the edge device(s) 1506 and the cloud server 1508 are ML proxies addressed by the message, the edge device(s) 1506 and the cloud server 1508 set up the channel with their own ML service entity. According to step 9, if the edge device(s) 1506 and the cloud server 1508 are not ML proxies addressed by the message, the edge device(s) 1506 and the cloud server 1508 forward the received ML service channel establishment message to the addressed ML proxies.

[0137] In response to the ML service channel establishment layer 1530 performing its functionality, the ML application data is actually processed via the ML application data layer 1540. Specifically, according to step 10, the ML application on the IoT device(s) 1502 exchanges data input and data output (taking some data as parameters, passing some data back as output) for a given ML service with the gateway agent(s) 1506 to which it is communicatively coupled. According to step 11, if the logical channel for the ML service is established with the local ML service (e.g., via the ML service channel establishment 1530), then the gateway device(s) 1504 exchanges application data input and data output with the IoT device(s) 1502. According to step 12, if the logical channel for the ML service is not established with the local ML service, then the gateway device(s) 1504 proxies or routes the ML application data input and data output to the remote ML agent through the edge device(s) 1506 and cloud server 1508 routing network.

[0138] According to step 13, if the logical channel for ML service is established with its local ML service, the edge device(s) 1506 and cloud server 1508 ML agents exchange their application data input and data output with the gateway device(s) 1506. According to step 14, if the logical channel for ML service is not established, the edge device(s) 1506 and cloud server 1508 proxy or route the ML application data input and data output to the channel remote gateway device(s) 1504 to which it is communicatively coupled. According to step 15, if the logical channel is established with the local ML service, the gateway device(s) 1504 exchange the ML application data input and data output with the remote gateway ML agent through the edge device(s) 1506 routing network.

[0139] Fig.16 is a block diagram illustrating how ML requests are routed from a client to a destination client device according to some embodiments. As described herein, ML processing resources (i.e., ML capabilities and computer resource characteristics) are distributed throughout a network and optimally shared among all ML clients residing within the ML network. Optimal sharing is achieved by routing ML requests from ML clients to the most appropriate ML agent or network device hosting the ML resources for processing the request. In some embodiments, such routing decisions are made according to routing policies configured in ML agents along the data path from the ML client to the destination ML agent hosting the corresponding ML resources.

[0140] Fig.16 For example, an intermediate proxy 1606 is illustrated between the ML proxy 1604 directly connected to the requesting ML client 1602 and the destination ML proxy 1609. Specifically, Fig.16 Requesting ML client 1602 (e.g., IoT device or client device 102) illustrating that ML agent discovery data storage device 1602-1 has been read routes ML request 1603 to directly connected ML agent 1604 (e.g., network device 116). Agent 1604 may use routing policy 1604-1 to route request 1605 to intermediate ML agent 1606, which may use its routing policy 1606 to route ML request 1607 to destination ML agent 169.

[0141] The "best route" and "routing policy" for routing described herein may refer to any suitable policy described herein, such as a policy based on ML learning capabilities and / or computer resource characteristics (i.e., ML resources). In some embodiments, a default routing policy is assigned to the ML agent during initialization (e.g., ML service advertisement) and may be dynamically configured during operation according to a number of factors, including network traffic status and the operational status of the ML agent (e.g., ML processing load (e.g., in terms of CPU and memory consumption)). In some embodiments, the routing policy takes into account optimization factors such as the following (all of which may be included in the routing resource list 120): latency for ML processing and data transfer (required by ML clients); ML processing capacity (e.g., supported ML models and model performance); priority of ML requests, which is assigned per ML client or per ML application; QoS requirements of the ML application (including real-time, latency, data rate requirements); and ML input and output locations within the network.

[0142] In some embodiments, routing decisions are made by Figure 1 The routing determiner 104 of FIG. 1604 determines the routing decision. In some embodiments, the routing decision is made hop by hop starting at the direct ML proxy (e.g., 1604) to which the requesting ML client is connected. This proxy decides whether it can handle the request from the ML client according to its configured routing policy. If not, it selects another ML proxy (e.g., 1606) to which it is directly connected, again according to the configured routing policy, and forwards the request to the selected ML proxy. Such chained decisions propagate toward the destination ML proxy (e.g., proxy 1609) that will handle the original request.

[0143] The ML request (e.g., 1603) from the ML client contains information about the requesting ML application. This information enables application-dependent routing decisions, as described above. In some embodiments, when a request is forwarded from one hop to the next, it may include any appropriate routing information that reflects the cumulative routing decisions made so far, so that the next hop can take this information into account when making its own routing decision. As an example, the original ML request 1603 from the ML client 1602 may contain a latency-constrained budget required by the ML application. In this case, each ML agent determines the latency for its internal processing and the latency for transmitting the request to the next hop of its choice. It then infers these latency from the latency-constrained budget in the ML request before transmitting the request to the next hop. Another example of routing information added to the ML request is the routing addresses (e.g., IP addresses, port numbers, and domain names) of all network devices along the end-to-end route from the requesting ML client to the destination ML agent 1609. This information can be used to route responses from the destination ML agent 1609 back to the requesting ML client 1602.

[0144] Fig.17 is a block diagram illustrating ML network (a) and ML network (b) in a wired broadband access network infrastructure according to some embodiments. Fig.17 It is illustrated that Internet Protocol Set Top Boxes (IP STBs) 1711 and 1721 are ML clients 1709 and 1719 respectively (eg, client device 102), while cable modems 1707 and 1717 and remote PHY devices (RPDs) 1705 and 1715 are ML agents (eg, network devices 116, 112, and 106).

[0145] Assuming that there are two MML applications in the IP STB, the following information is included in the ML request for the ML client 1709: (1) the budget for the total processing and communication delay incurred by the ML agent, denoted by L; and (2) the required ML model, denoted by Ma. If it is assumed that the ML agent in the cable modem 1707 and / or 1717 only supports model Ma, and the total processing delay for this model in the cable modem is Lcm<L, then the cable modem itself can service the request from the ML client 1709 without forwarding the request to any other agent.

[0146] For ML client 1719 (corresponding to ML resource network (b)), the following information is included in the ML request: (1) the budget for the total processing and communication delay incurred by the ML agent, denoted by L; and (2) the required ML model, denoted by Mb. In various embodiments, it is assumed that the ML agent within RPD 1715 supports model Mb, and the processing delay within RPD 1715 and the communication delay between cable modem 1717 and RPD 1715 are Lrpd and Lcom, respectively. In addition, in some embodiments, it is assumed that Lcm+Lrpd+Lcom<L. In this case, the cable modem 1717 itself cannot service the request from the ML client 1719 because it does not support model Mb. In some embodiments, through the ML resource discovery process as described herein (e.g., Fig.14 1420), making the ML agent within the cable modem 1717 aware that model Mb is supported at the directly connected (e.g., directly attached) RPD. Therefore, the cable modem 1717 forwards the request for this application to RPD 1715, where the included delay budget is decremented by Lcom. In some embodiments, the contact address (e.g., IP address) of the cable modem 1717 is also included in the request. After receiving the ML request, RPD 1715 concludes that it can service the request because it supports M and Lcm+Lrpd+Lcom <L。

[0147] Fig.18 1800 is a block diagram of an example ML agent device 1800 according to some embodiments. In various embodiments, ML distributed processing protocol messages 1801 sent by an ML agent device (or client device) to the ML agent device 1800 (and ML client devices) via a network medium are encapsulated into IoT transmission data units by an IoT protocol entity 1805 running on a CPU 1823. These data units are (optionally encrypted and) encapsulated by a network transmission service driver (not shown) of a network interface 1803 into network protocol data units sent to a network interface controller (not shown) of a network I / F 1803. In some embodiments, ML distributed processing protocol messages received from other ML agents and ML client devices via a network medium are (optionally decrypted and) presented to the IoT protocol entity 1805 through a network socket interface.

[0148] In some embodiments, the IoT protocol entity 1805 unpacks these data units to retrieve the data payload, which may be an ML distributed processing control message (e.g., a message for routing a specific request to a specific network device) and / or ML application input data (e.g., a video stream of an object). If the payload data is ML application input data, it is stored in the data RAM 1815 for pre-processing by the ML pre-processor 1817 before being presented to the ML core(s) 1819 for ML processing. The ML application output data 1821 is created by the ML core(s) 1819, post-processed by the ML post-processor 1815, and stored in the data RAM 1813, and then packaged by the IoT protocol entity 1805 for transmission, as described above.

[0149] Specifically, ML application input data 1801 is sent to the NIC buffer 1809 and socket buffer 1811 of the network I / F 1803. The network interface card (NIC) buffer is a memory storage area configured to store data packets waiting to be transmitted over the network or to store data packets received over the network. The socket buffer is a memory storage area or structure that controls and stores state information of the socket layer. The request 1801 is responsively routed to the IoT protocol buffer 1813. The IoT protocol buffer is a memory storage area or structure that holds data ready for processing by the machine learning core or temporarily stores the data. In response, the request 1801 is routed to the M1 pre-processing 1817 (e.g., Figure 5 514), and then, is routed to (several) machine learning cores 1819. In response to processing by (several) machine learning cores 1819, the processed data goes to post-ML process 1815 (e.g. Figure 8810), and the resulting ML application output data 1821 is routed to another network device or routed back to the client device.

[0150] Fig.19 is a block diagram of an example client device according to some embodiments. Fig.19 A client request (i.e., ML application input data 1909) illustrating a request for one or more machine learning processes to be performed by one or more network devices is sent from a client device 1900. In response to the request being routed and processed by the network device, data (i.e., ML application output data 1911) is provided back to the client device 1900.

[0151] Specifically, ML application input data 1909 from the ML application running on the CPU 1907 is sent by the ML client device 1900 to the ML agent device (e.g., agent device 1800) via the network medium through the ML distributed processing protocol 1905. These data are encapsulated into IoT transmission data units by the IoT protocol entity 1905 running on the CPU 1907 together with the ML distributed processing protocol message. These data units are encapsulated by the network transmission service driver (optionally encrypted and) into network protocol data units sent to the network interface controller through the network socket interface 1903. The ML distributed processing protocol data received from other ML agent devices via the network medium is (optionally decrypted and) presented to the IoT protocol entity through the network socket interface 1903.

[0152] ML application output data 1911 is first received and processed through NIC buffer 1913 and socket buffer 1915. IoT protocol entity 1905 unpacks these data units to retrieve the data payload, which may be ML distributed processing control messages (e.g., control signals) or ML application data (e.g., detected objects and amplification data).

[0153] If the payload data is ML application data, it is stored in the data RAM 1919 for presentation to the ML application. The IoT protocol entity 1905 unpacks these data units to retrieve the data payload, which may be an ML distributed processing control message or ML application data. If the payload data is ML application input data, it is stored in the data RAM 1919 for pre-processing by the ML pre-processor before presentation to the ML core for ML processing.

[0154] When the payload is an ML distributed processing control message, it is sent to an IoT actuator 1921 which causes some tangible action on the device via a control signal. The IoT actuator takes electrical input and turns it into a physical action. For example, the IoT actuator 1921 can physically set the temperature on a smart thermostat, physically turn on / off a light, cause a TV to turn on / off, turn on / off a washing machine, cause a smart speaker to emit a sound, and the like.

[0155] Fig. 20 2000 is a flow chart of an example process 2000 for generating or updating a data object with data of computer resource characteristic information and machine learning capability information of a specified network device according to some embodiments. In some embodiments, process 2000 represents a client request being processed (e.g., via Fig.21 In some embodiments, process 2000 represents an online or runtime process performed in response to a client request. In some embodiments, process 2000 is a combination of online and offline processes. For example, at block 2004, machine learning capability information may be received offline before a client request and computer resource characteristic information are received in response to a client request to determine real-time (or near real-time) computer resource characteristics of a particular network device.

[0156] Process 2000 (and / or any of the functionality described herein (e.g., process 2100)) may be performed by processing logic including hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions running on a processor to perform hardware simulation), firmware, or a combination thereof. Although specific blocks described in the present disclosure are referenced in a specific order and in a specific number, it should be understood that any block may occur substantially in parallel with any other block or before or after any other block. In addition, there may be more (or fewer) blocks than illustrated. Such added blocks may include blocks embodying any functionality described herein. Computer-implemented methods, systems, and / or computer program products / computer-readable storage media as described herein may perform or cause to perform processes 2000 and / or 2100 or any other functionality described herein.

[0157] According to block 2002, a network device broadcasts its own computer resource characteristic information and machine learning capability information to one or more other network devices. Fig.14 ML service discovery layer 120 steps and Fig.15 The ML service discovery advertisement layer 1520 is described. For example, Fig.14 The GW device 1414 transmits the “ML service advertisement” to the edge device 1412 in step 1. In another example, Figure 1The network device 116 may broadcast its child resource list 120 - 3 to other network devices 106 and 112 so that these other network devices 106 and 112 have ML resource information about the network device 116 .

[0158] According to block 2004, the network device receives computer resource characteristic information and machine learning capability information from one or more other network devices. Fig.14 ML service discovery layer 120 steps and Fig.15 ML service discovery advertisement layer 1520 is described. For example, in step 3, GW device 1414 may receive computer resource characteristic information and machine learning capability information of edge device 1412 from edge device 1412. In another example, network device 116 may receive sub-lists 120-1 and 120-2 from network devices 112 and 106, respectively.

[0159] According to block 2006, the network device generates and / or updates a data object (e.g., a list or other data structure) that specifies one or more network devices and computer resource characteristic information and machine learning capability information of the network devices. Fig.14 ML service discovery layer 120 steps and Fig.15 The ML service discovery advertisement layer 1520 is described. For example, Fig.14 The “add” step of may include this functionality (eg, step 5 of “add (ML_i) to ML service database”). In another example, the final resource list 120 may correspond to an updated list, where sub-lists 120-1 and 120-2 are updates.

[0160] Fig.21 is a flow chart of an example process 2100 for executing a client request to process data via one or more machine learning processes according to some embodiments. In some embodiments, process 2100 represents runtime or online functionality and occurs at Fig. 20 The process started after 2000.

[0161] Per block 2103, a client request is received to process data via one or more machine learning processes. The client request may originate from a client device (e.g., an IoT device). In an illustrative example, a network device may receive a transmitted client request that is a request to detect and follow an object in a video stream. In some embodiments, block 2103 includes establishing an ML service channel between the client device and the network device(s). Examples of this are described with respect to ML service channel creation 1430 and / or ML service channel establishment steps at ML service channel establishment layer 1530. In an illustrative example of block 2103, at Fig.14After step 7 of block 2103, GW device 1406 receives the ML service request to establish a connection with GW device 1406, and all other channels are opened via steps 8, 9, 10, and 11. In another example of block 2103, steps 8, 9, 10, and / or 11 may occur.

[0162] Alternatively or in addition, block 2103 may include receiving the actual payload of the request (e.g., a request to detect and follow an object in a video stream) after a channel has been established between the client device and the network device(s). Examples of these embodiments include Fig.14 The receiving step performed by the ML agent device in the ML application data 1440 layer (for example, the GW device 1406 receives the ML application data transmitted by the IoT device 1402 in step 12) and Fig.15 1500 at the receiving step at the ML application data layer 1540 (e.g., the exchange of data input and data output at steps 13 and 14). Other examples include when network device 1800 receives ML application input data 1801 or when network device 116 receives request 150 from client device 102.

[0163] In some embodiments, the client device transmitting the request at block 2103 is an Internet of Things (IoT) device and / or an Internet of Everything (IoE) device. For example, the client device may be a connected appliance (e.g., a smart washing machine, a smart dishwasher, a smart thermostat), a smart home security system, a smart television, a wearable health monitor, a smart light system, a person (e.g., a smart phone), a car, and the like.

[0164] According to block 2105, a first set of attributes associated with a first network device and a second set of attributes associated with a second network device are determined. The first set of attributes and the second set of attributes include computer resource characteristics of the respective network devices and / or machine learning processing capabilities of the respective network devices. For example, Figure 1 Network device 116 may determine these characteristics of itself and network device 112 by reading sub-lists 120-1 and 120-3. In some embodiments, block 2105 occurs in response to block 2103.

[0165] In some embodiments, block 2105 alternatively or additionally "compares" at least a first portion of the first set of attributes to at least another portion of the second attributes. For example, in response to receiving the request at block 2103, the client device may compare the attributes of the network devices to determine where to route the request based on which network device is more computationally resource efficient or has more capable machine learning functionality to service the request. In an illustrative example, Figure 1The network device 116 can compare the real-time memory capacity, CPU or other computer characteristics of the network device 112 (via the sub-resource list 120-1) with its own computer characteristics (via the sub-resource list 120-3). In another illustrative example, Figure 1 The network devices 116 of the different network devices may compare the ML capabilities (e.g., ML model type hyperparameters, etc.) of the different network devices. Alternatively or in addition, block 2015 may include "comparing" the payload of the request with the properties of each network device (e.g., resource list 120) to determine where to route the request.

[0166] In some embodiments, the first network device is located at the first layer of the hierarchical computer network (eg, Figure 1 N), wherein the first network device has a first machine learning core (e.g. Figure 1 ML core 114). For example, returning to reference Figure 2 , the first layer can be the "edge layer" (or Figure 4 The first network device may be an edge device 209 that exists within a broadband access network as an example of a hierarchical network. In some embodiments, the second network device is located in at least one of the first layer or the second layer of the hierarchical computer network, wherein the second network device has a second machine learning core. For example, the second layer may be Figure 2 The second network device may be Figure 2 As an alternative to these examples, the layer or network device may be about Figure 1 , Figure 2 or any suitable layers and network devices described in Figure 3. As described herein, the terms "first" and "second" layers or network devices do not indicate a processing order, but rather indicate that they are separate or distinguishable layers or devices.

[0167] In some embodiments, the one or more computer characteristics include one or more of data transfer rate (e.g., megabits per second), processing capacity (the number of operations a processor can perform in a set amount of time), transport protocol (e.g., TCP / IP, UDP, IoT, etc.), latency (e.g., round trip time (RTT) or time to first byte (TTFB)), bandwidth, memory capacity or availability, and security. Security refers to security attributes or capabilities employed by a network device (e.g., whether the network device employs encryption, obfuscation, types of authentication, types of malware that the network device can detect, etc.).

[0168] In some embodiments, one or more machine learning capabilities include one or more of the types of machine learning models hosted on the corresponding network device (e.g., CNN, random forest, long short-term memory (LSTM), recurrent neural network (RNN), Gaussian mixture model (GMM), BERT, WORD2vec, supervised learning, unsupervised learning, classifier, clustering model, regression model, etc.), hyperparameters used in the corresponding machine learning model, input data pre-processing functions, output data post-processing functions, and data paths for input and output data streams.

[0169] In machine learning, a hyperparameter is a parameter whose value is used to control the learning process. By contrast, other parameter values ​​(usually node weights) are derived through training. Hyperparameters can be classified as model hyperparameters, which cannot be inferred when matching machines to training groups, because they refer to model selection tasks or algorithmic hyperparameters, which in principle have no effect on the performance of the model but affect the speed and quality of the learning process. Examples of model hyperparameters are the topology and size of the neural network. Examples of algorithmic hyperparameters are learning rate and mini-batch size. "Data path" refers to the hardware (functional units) that perform all required operations, such as the ALUs, registers, and internal buses within a CPU.

[0170] "Input preprocessing functions" transform the raw data into a clean dataset ready for processing by a machine learning model. These functions may indicate whether the preprocessing is done by or its type: Get dataset, Import all key libraries, Import dataset, Identify and handle missing values, Encode categorical data, Split dataset, Indicate whether the preprocessing uses a binary threshold to transform the data. All values ​​above the threshold are labeled as 1 and all values ​​equal to or below the threshold are labeled as 0, Indicate data standardization and how feature scaling or normalization is performed. "Output data post-processing" functions typically include the following types or methods: various pruning routines, rule quality processing, rule filtering, rule combination, model combination, or even knowledge integration. Output data post-processing acts as a filter for noisy and imprecise knowledge derived by the machine learning model.

[0171] In some embodiments, the first network device and the second network device are at least one of the following: an edge computing device (eg, Figure 4 424), cloud nodes (such as 420), home gateways, network switches, phones, set-top boxes, televisions, home network adapters, and Internet access gateways.

[0172] According to block 2107, a first network device is selected to route the client request to. In some embodiments, block 2107 is based at least in part on the client request and the determination of the first set of attributes and the second set of attributes made at block 2105. For example, the client request may specify a particular object that it needs to detect in the video stream. In response, Figure 1 Network device 116 may determine, via resource list 120, that the only network device with object detection functionality is network device 112. Therefore, network device 112 is selected to route the request to.

[0173] In some embodiments, block 2107 is based at least in part on a comparison, as described above with respect to block 2104. For example, referring back to Figure 1 , routing determiner 104 can compare the CPU and memory of network device 112 to the CPU and memory of network device 106 and compare their machine learning capabilities via resource list 120. Based on network device 112 having greater memory availability, less CPU utilization, and / or a more suitable machine learning model, for example, network device 112 is selected to route the client request to it.

[0174] Some embodiments further avoid selecting a second network device for routing. Figure 1 , network device 116 may categorically avoid selecting network device 106 for routing based on a comparison or determination that its machine learning capabilities and / or computer resource characteristics are not well suited to serving the client request. Alternatively, some embodiments select both network devices to serve the client request. For example, a first network device may be selected to process information through a first machine learning model, and a second network device may be selected to process other information through a second machine learning model.

[0175] Per block 2109, certain embodiments cause the first network device to process the request. In some embodiments, block 2109 is based at least in part on the selection at block 2107. In some embodiments, block 2109 includes or is preceded by actually routing the client request to the first network device based on the selection at block 2107.

[0176] In some embodiments, processing of the request of block 2109 includes detecting abnormal activity at the first network device and via one or more machine learning models hosted at the first network device. Figure 6 Description, which describes, among other things, diagnosis of connectivity problems, detection of media content leakage, and / or detection of malicious network intrusions.

[0177] In some embodiments, processing of the request of block 2109 includes detecting device-specific and network-wide physical layer issues at the first network device and via one or more machine learning models hosted at the first network device. Figure 7 Description, which describes the detection and classification of network failures, characterization of sub-network conditions, and the like.

[0178] In some embodiments, processing of the request of block 2109 includes detecting, at the first network device and via one or more machine learning models hosted at the first network device, one or more objects in the one or more images. Fig. 9 and / or Fig.10 Description, for example, describes detecting and tracking an object, and then, zooming in on the object.

[0179] In some embodiments, at least in part in response to the routing or processing at block 2109, some embodiments route the request to a third network device, wherein the first network device, the second network device, and the third network device are positioned in a hierarchical network, and wherein the first network device and the second network device are positioned in a first layer of the hierarchical network, and the third network device is positioned in a second layer of the hierarchical network, the second layer being further downstream relative to the first layer in the hierarchical network. Being "downstream" means that a layer is configured to process the request at a later time relative to another layer. This example of a hierarchical network is about Figure 2 and Figure 4 For example, about Figure 2 , Figure 2 The set-top box may first process the client request at the CPE layer, and the layer at the top, namely the edge layer, may then process the request via the edge node 209.

[0180] In some embodiments, a first portion of the data processed at block 2109 is processed at a first network device and a second portion of the data is processed at a third network device. For example, as described above, multiple network devices may be used to service client requests. For example, the first portion may be video stream data to be processed for object detection, and the second portion may be packet data to be inspected for malware or other malicious activity.

[0181] Per block 2111, some embodiments cause data to be transmitted to the client device. In some embodiments, this is at least partially responsive to the processing of the client request at block 2109. For example, in response to the first network device detecting and tracking the object via the client request, an embodiment causes transmission of the detected and magnified object (e.g., causing a visible bounding box to appear over the object and magnifying pixels of the magnified image). In some embodiments, the transmission of data includes transmitting, for example, information about Figure 6 , Figure 7 , Figure 8 , Fig. 9 and / or Fig.10 An alert of any detection made is transmitted to the client device. For example, an alert may be an indication or notification that unusual activity is transmitted (e.g., a natural language message that malware has been detected).

[0182] Embodiments of the present disclosure may be described in the general context of computer code or machine-usable instructions (e.g., program modules) that include computer-usable or computer-executable instructions executed by a computer or other machine (e.g., a personal data assistant, a smart phone, a tablet PC, or other handheld device). Typically, a program module that includes routines, programs, objects, components, data structures, and the like refers to code that performs a specific task or implements a specific abstract data type. Embodiments of the present disclosure may be practiced in various system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, and the like. Embodiments of the present disclosure may also be practiced in a distributed computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including memory storage devices.

[0183] refer to Fig. 22 , computing device 2200 includes bus 10 that directly or indirectly couples the following devices: memory 12, one or more processors 14, one or more presentation components 16, one or more input / output (I / O) ports 18, one or more I / O components 20, and an illustrative power supply 22. Bus 10 represents something that can be one or more buses, such as an address bus, a data bus, or a combination thereof. Although for clarity, bus 10 is not intended to be confused with bus 10, as shown in FIG. 1 , bus 10 is not intended to be confused with bus 10. Fig. 22 The various blocks of the invention are shown with lines, but in reality, these blocks represent logic and not necessarily actual components. For example, one may consider presentation components, such as display devices, as I / O components. Also, the processor has memory. The inventors recognize that this is the nature of the art and reiterate that Fig. 22 The figures are merely illustrative of exemplary computing devices that may be used in conjunction with one or more embodiments of the present disclosure. No distinction is made between categories such as "workstation," "server," "laptop," "handheld device," etc., since each is considered to be a separate entity in the same class. Fig. 22 within the scope of and with reference to a “computing device”.

[0184] The computing device 2200 typically includes a variety of computer-readable media. Computer-readable media may be any available media that can be accessed by the computing device 2200 and includes both volatile and non-volatile media, removable and non-removable media. By way of example (and not limitation), computer-readable media may include computer storage media and communication media. Computer storage media include both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include (but are not limited to) RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage devices, cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other media that can be used to store desired information and can be accessed by the computing device 2200. Computer storage media do not include the signal itself. Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal (such as a carrier wave or other transmission mechanism) and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example (and not limitation), communication media includes wired media such as a wired network or direct-wired connection and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0185] Memory 12 includes computer storage media in the form of volatile and / or non-volatile memory. Memory may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. Computing device 2200 includes one or more processors 14 that read data from various entities such as memory 12 or I / O components 20. Presentation component(s) 16 present data indications to a user or other device. Other examples of presentation components may include display devices, speakers, printing components, vibration components, and the like.

[0186] I / O ports 18 allow computing device 2200 to be logically coupled to other devices, some of which may be built-in, including I / O components 20. Illustrative components include microphones, joysticks, game consoles, satellite dishes, scanners, printers, wireless devices, etc. I / O components 20 may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some examples, the input may be transmitted to an appropriate network element for further processing. The NUI may implement any combination of voice recognition, touch and stylus recognition, facial recognition, biometric recognition, gesture recognition on and adjacent to the screen, air gestures, head and eye tracking, and touch recognition associated with a display on computing device 2200. Computing device 2200 may be equipped with a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, and combinations of these for gesture detection and recognition. In addition, computing device 2200 may be equipped with an accelerometer or gyroscope to implement motion detection. The output of the accelerometer or gyroscope may be provided to a display of computing device 2200 to render immersive augmented reality or virtual reality.

[0187] Some embodiments of the computing device 2200 may include one or more radios 24 (or similar wireless communication components). The radio 24 transmits and receives radio or wireless communications. The computing device 2200 may be a wireless terminal adapted to receive communications and media via various wireless networks. The computing device 2200 may communicate with other devices via wireless protocols such as Code Division Multiple Access (“CDMA”), Global System for Mobile (“GSM”), or Time Division Multiple Access (“TDMA”), among others. Radio communications may be short-range connections, long-range connections, or a combination of both short-range and long-range wireless telecommunication connections. When we refer to “short” and “long” type connections, we do not mean to refer to the spatial relationship between two devices. Instead, we generally refer to short-range and long-range as different categories or types of connections (i.e., primary connections and secondary connections). Short-range connections may include (by way of example (and not limitation)) connections to devices (such as mobile hotspots) that use the 802.11 protocol to provide access to a wireless communication network (such as a WLAN connection). A Bluetooth connection to another computing device is a second example of a short-range connection or a near field communication connection. Long-range connections may include connections using, by way of example (and not limitation), one or more of CDMA, GPRS, GSM, TDMA, and 802.16 protocols.

[0188] As used herein, the term "group" may be used to refer to an ordered (i.e., sequential) or unordered (i.e., non-sequential) collection of objects (or elements), such as (but not limited to) data elements (e.g., events, cells, rows, and the like). A group may include N elements, where N is any non-negative integer of 1 or greater. That is, a group may include 1, 2, 3 ... N objects and / or elements, where N is a positive integer with no upper limit. A group may include only a single element. In other embodiments, a group may include a number of elements significantly greater than 1, 2, or 3 elements. For example, the various embodiments and examples described herein refer to a "group" of features or a "group" of feature vectors. This refers to one or more features and one or more feature vectors, respectively.

[0189] Many different arrangements of the various components depicted, as well as components not shown, are possible without departing from the scope of the claims below. Embodiments of the present disclosure have been described, intended to be illustrative and non-limiting. It will become apparent to the reader of the present disclosure after and as a result of reading the alternative embodiments. Alternative ways of implementing the foregoing may be accomplished without departing from the scope of the claims below. Certain features and subcombinations are of utility and may be employed without reference to other features and subcombinations and are considered within the scope of the claims.

Claims

1. A computer-implemented method comprising: receiving a client request to process data via one or more machine learning processes, the client request being associated with a client device; comparing a first set of attributes associated with a first network device and a second set of attributes associated with a second network device, the first set of attributes and the second set of attributes comprising one or more computer resource characteristics of the respective network devices and one or more machine learning processing capabilities of the respective network devices; based at least in part on the comparison, selecting the first network device to route the client request to the first network device and avoiding selecting the second network device; Based at least in part on the selection, causing the first network device to process the client request; and causing data to be transmitted to the client device at least in part in response to the processing of the client request; wherein the first network device includes a first machine learning core that implements end-to-end processing associated with an object, wherein the object represents any logical entity having a unique signature embedded in ML input data, and wherein the first machine learning core is configured to: detecting and classifying objects present in the ML input data by detecting signatures associated with the objects, continuously or periodically tracking the object across a stream of the ML input data, zooming in on the object by scaling its associated attributes or detecting additional detailed attributes, and The data is post-processed by generating results from the detection, tracking and scaling and transmitting notifications applying specific data privacy rules.

2. The method of claim 1, wherein the one or more computer resource characteristics include at least one characteristic from a group of characteristics consisting of: data transfer rate, processing capacity, transmission protocol, latency, bandwidth, memory capacity, neural network type or neural network structure, and security.

3. The method of claim 1, wherein the one or more machine learning processing capabilities include at least one capability from a group of capabilities consisting of: a machine learning model type hosted on a corresponding network device, hyperparameters used in the corresponding machine learning model, input data preprocessing functions, output data post-processing functions, and data paths for input and output data streams.

4. The method of claim 1, wherein the first network device and the second network device comprise at least one network device from a group of network devices consisting of: an edge computing device, a cloud node, a home gateway, a network switch, a phone, a set-top box, a television, a home network adapter, and an Internet access gateway. The method of claim 4 , wherein the client device and the first network device or the second network device are modules located on the same device.

6. The method of claim 1, wherein the processing of the client request includes detecting anomalous activity at the first network device and via one or more machine learning models hosted at the first network device.

7. The method of claim 1 , wherein the request is preceded by an initial control request sent to a carrier cloud service, and wherein the first network device and the second network device are configured by the cloud service to support ML processing of the client device, and wherein the data for the ML processing is sent directly to the first network device.

8. The method of claim 7, wherein processing results of the ML processing of the data are: sent to the client device for privacy reasons, sent to the cloud service for network or client device diagnostics, and sent to both the cloud service and the client device for service purposes.

9. The method of claim 1, wherein the processing of the client request includes detecting, at the first network device and via one or more machine learning models hosted at the first network device, one or more objects in one or more images.

10. The method of claim 1, further comprising routing the request to a third network device at least in part in response to the routing, wherein the first network device, the second network device, and the third network device are positioned in a hierarchical network, and wherein the first network device and the second network device are positioned in a first layer of the hierarchical network, and the third network device is positioned in a second layer of the hierarchical network, the second layer being further downstream relative to the first layer in the hierarchical network. 11 . The method of claim 10 , wherein a first portion of the data is processed at the first network device and wherein a second portion of the data is processed at the third network device.

12. A system for processing data, comprising: a client device configured to transmit a client request to process data via one or more machine learning processes; A first network device positioned in a first layer of a hierarchical computer network, the first network device having a first machine learning core; and a second network device positioned in at least one of the first layer or the second layer of the hierarchical computer network, the second network device having a second machine learning core, the second network device being configured to determine a first set of attributes associated with the first network device and to determine a second set of attributes associated with the second network device, the first set of attributes and the second set of attributes comprising one or more computer resource characteristics of the respective network device and one or more machine learning processing capabilities of the respective network device; The second network device is further configured to: selecting the first network device to process the client request based at least in part on the determination of the first set of attributes and the second set of attributes, and Based on the selection, routing the client request to the first network device; wherein the first machine learning core implements end-to-end processing associated with an object, wherein the object represents any logical entity having a unique signature embedded in ML input data, and wherein the first machine learning core is configured to: detecting and classifying objects present in the ML input data by detecting signatures associated with the objects, continuously or periodically tracking the object across a stream of the ML input data, zooming in on the object by scaling its associated attributes or detecting additional detailed attributes, and The data is post-processed by generating results from the detection, tracking and scaling and transmitting notifications applying specific data privacy rules.

13. The system of claim 12, wherein the selection is further based on one or more computer resource characteristics of the first network device and the second network device.

14. A system according to claim 12, wherein the one or more machine learning processing capabilities include at least one capability from a group of capabilities consisting of: a machine learning model type hosted on a corresponding network device, hyperparameters used in the corresponding machine learning model, input data pre-processing functions, output data post-processing functions, and data paths for input and output data streams.

15. The system of claim 12, wherein the first network device and the second network device comprise at least one network device from a group of network devices consisting of an edge computing device, a cloud node, a home gateway, a network switch, a phone, a set-top box, a television, a home network adapter, and an Internet access gateway.

16. The system of claim 12, wherein the client device is an Internet of Things (IoT) device or an Internet of Everything (IoE) device.

17. The system of claim 12, wherein the first network device is further configured to transmit data to the client device in response to the processing of the client request, wherein the transmission of the data comprises transmitting an alert of abnormal activity to the client device.

18. One or more computer storage media storing computer usable instructions that, when used by one or more processors, cause the one or more processors to perform a method comprising: receiving a client request to process data via one or more machine learning processes, the client request originating from a client device; determining a first set of attributes associated with a first network device and determining a second set of attributes associated with a second network device, the first set of attributes and the second set of attributes comprising one or more machine learning processing capabilities of the respective network devices; based at least in part on the client request and the determination of the first set of attributes and the second set of attributes, selecting the first network device and the second network device to route the client request to; and Based at least in part on the selection, causing the first network device to process a first portion of the data and causing the second network device to process a second portion of the data; wherein the first network device includes a first machine learning core that implements end-to-end processing associated with an object, wherein the object represents any logical entity having a unique signature embedded in ML input data, and wherein the first machine learning core is configured to: detecting and classifying objects present in the ML input data by detecting signatures associated with the objects, continuously or periodically tracking the object across a stream of the ML input data, zooming in on the object by scaling its associated attributes or detecting additional detailed attributes, and The data is post-processed by generating results from the detection, tracking and scaling and transmitting notifications applying specific data privacy rules.

19. The one or more computer storage media of claim 18, wherein the selection is further based on one or more computer resource characteristics of the first network device and the second network device.

Images