Machine learning agent system

Abstract

The invention relates to a machine learning agent system. A system is provided that includes a machine learning ML application client configured to monitor a data stream communicated with a connected device for application data associated with the ML application client, and capture the associated application data. An ML proxy client is configured to receive an application request from the ML application client, wherein the application request includes the application data captured by the ML application client; sending a proxy request to an ML proxy server, wherein the proxy request includes the application data and the indicated reasoning operation to be performed on the application data; receiving a proxy response from the ML proxy server, wherein the proxy response includes a result of the reasoning operation performed on the application data; and sending a report based on the proxy response to a controller external to the first network.

Description

Technical Field

[0001] This description generally refers to service provider data networks, including, for example, the use of machine learning applications across service provider data networks. Background Technology

[0002] Service provider data networks can be used to deliver services such as video content, internet access, telephone, and gaming to subscribers. Service providers are increasingly using applications that incorporate machine learning operations for tasks such as network monitoring and performance evaluation, troubleshooting, and improving customer experience. Furthermore, machine learning operations are used to support and enhance the capabilities of Internet of Things (IoT) devices. In a cloud-based approach, data can be collected by devices at various locations within the service provider's data network and sent to the cloud for processing, including machine learning operations. However, this cloud-based approach can be inefficient in utilizing network bandwidth and storage capacity, may increase latency for applications using machine learning operations, and may expose sensitive information in the data to parties outside the service provider's data network. Summary of the Invention

[0003] In one aspect, this disclosure relates to an electronic device comprising: a first network interface configured to transmit and receive communications on a first network; a second network interface configured to transmit and receive communications on a second network different from the first network; a computer-readable storage medium storing one or more instruction sequences; and a processing circuitry configured to execute the one or more instruction sequences to perform operations including: receiving an application request from a machine learning (ML) application client device on the first network, wherein the application request includes application data captured by the ML application client device from one or more data streams transmitted via the first network and the second network and a connected device on the first network; sending a proxy request to an ML proxy server device on the first network, wherein the proxy request includes the application data and an indicated inference operation to be performed on the application data; receiving a proxy response from the ML proxy server device, wherein the proxy response includes the result of the inference operation performed by the ML proxy server device on the application data; and sending a report based on the proxy response to a controller outside the first network via the second network interface.

[0004] In another aspect, this disclosure relates to a method comprising: receiving an application request from a machine learning (ML) application client device on a first network, wherein the application request includes application data captured by the ML application client device from one or more data streams communicated with a connected device on the first network and an indicated inference operation to be performed on the application data; preprocessing the application data based on parameters associated with the indicated inference operation; sending a proxy request to an ML proxy server device on the first network, wherein the proxy request includes the preprocessed application data and the indicated inference operation to be performed on the preprocessed application data; receiving a proxy response from the ML proxy server device, wherein the proxy response includes the result of the inference operation performed by the ML proxy server device on the application data; and sending a report based on the proxy response to a controller on a second network outside the first network.

[0005] In another aspect, this disclosure relates to a system comprising: a machine learning (ML) application client configured to perform on a first device on a first network to: monitor one or more data streams transmitted via connected devices on the first network for application data associated with the ML application client; and capture the application data associated with the ML application client; an ML proxy server configured to perform on a second device on the first network to: initiate and perform ML inference operations; and an ML proxy client configured to perform on a third device on the first network to: receive an application request from the ML application client, wherein the application request includes application data captured by the ML application client from one or more data streams transmitted via connected devices; send a proxy request to the ML proxy server, wherein the proxy request includes the application data and an indicated inference operation to be performed on the application data; receive a proxy response from the ML proxy server, wherein the proxy response includes the result of the inference operation performed by the ML proxy server on the application data; and send a report based on the proxy response to a controller outside the first network. Attached Figure Description

[0006] Some features of this subject matter are set forth in the appended claims. However, for purposes of explanation, several aspects of this subject matter are illustrated in the following drawings.

[0007] Figure 1 Examples of network environments in which aspects of the technologies of this subject can be implemented are provided.

[0008] Figure 2Examples of routing devices based on aspects of the technology in this subject are described.

[0009] Figure 3 Examples of gateway devices based on aspects of the technology in this subject matter are described.

[0010] Figure 4 Examples of multifunctional electrical appliances (e.g., set-top boxes) based on aspects of the technology in this subject matter are provided.

[0011] Figure 5 This is a flowchart illustrating the operation of a machine learning agent system based on the technical aspects of this topic.

[0012] Figure 6 This is a block diagram illustrating the data flow associated with the operation of a machine learning application client, based on aspects of the technology in this subject matter.

[0013] Figure 7 This is a diagram illustrating the data flow associated with the operation of a machine learning agent client based on aspects of the technology in this topic.

[0014] Figure 8 This is a diagram illustrating the data flow associated with the operation of a machine learning agent server based on aspects of the technology in this subject.

[0015] Figure 9 This is a block diagram illustrating the data flow associated with the operation of machine learning application clients and machine learning agent clients according to the technical aspects of this topic.

[0016] Figure 10 This is a block diagram illustrating the data flow associated with the operation of machine learning agent clients and machine learning agent servers, based on aspects of the technology in this subject matter.

[0017] Figure 11 This is a block diagram illustrating the data flow associated with the operation of machine learning application clients, machine learning agent clients, and machine learning agent servers in an alternative configuration based on the technical aspects of this topic. Detailed Implementation

[0018] The detailed description set forth below is intended as a description of various configurations of the subject matter and is not intended to represent the only configuration in which the subject matter can be practiced. The accompanying drawings are incorporated herein and form part of the detailed description. The detailed description contains specific details for providing a thorough understanding of the subject matter. However, the subject matter is not limited to the specific details set forth herein and can be practiced without one or more of the specific details. In some instances, structures and components are shown in block diagram form to avoid obscuring the concepts of the subject matter.

[0019] Machine learning (ML) capabilities, such as neural network processing, are being added to many different types of devices. For example, ML systems (e.g., ML processing engines, neural processing units, hardware accelerators, etc.) are being implemented in customer premises equipment (CPEs), such as cable modems, set-top boxes, routers, etc. As these ML-enabled edge devices are widely deployed across service provider data networks, service providers are able to leverage these edge devices to facilitate and improve the use of ML applications to manage their data networks.

[0020] This subject matter provides a solution for utilizing the processing and computing resources of ML-enabled edge devices to serve ML applications running locally on a service provider's data network. According to aspects of this subject matter, the ML-enabled edge device can act as an ML agent for the device and the application running on a local area network (LAN) located at the customer's premises. For example, a cable gateway / modem can receive requests for ML operations from an ML application running on the LAN. Instead of sending the request and associated data outside the LAN to the cloud for processing, the cable gateway / modem can request an ML-enabled CPE device (e.g., a set-top box) operating as an ML agent server to process the ML operation. The ML agent server can then perform the requested ML operation (e.g., inference) and return the result to the cable gateway / modem. The cable gateway / modem can then report the result to an operator server in the cloud for the operator's use. In this way, using an ML agent server to process ML requests locally eliminates the need to send data outside the LAN, while still allowing the operator to utilize the ML application. Therefore, any sensitive information that may be included in the data remains on the LAN, and bandwidth / storage resources on the service provider's data network can be saved by keeping ML processing within the LAN. The following provides examples and descriptions of these and other features of the subject matter in more detail.

[0021] Figure 1 Examples of network environments 100 in which aspects of the subject matter can be implemented are illustrated. However, not all depicted components are necessary, and one or more embodiments may include additional components not shown in the figures. Changes may be made to the arrangement and type of components without departing from the spirit or scope of the claims set forth herein. Unless expressly stated otherwise, the connections or couplings (including electrical and communication connections and couplings) between the depicted or described components are not limited to direct connections or direct couplings, and may be implemented using one or more intermediate components.

[0022] Example network environment 100 includes a service provider data network 120 configured to provide services (e.g., video content, data, telephone, gaming, etc.) to local area networks (LANs) 140A to 140D operating at respective customer premises. Services may be provided within the service provider data network 120 or may involve resources communicated with the service provider data network 120 via a public communication network (e.g., the Internet 160). Communication within the service provider data network 120 may be provided using various types of transmission media, including, but not limited to, coaxial lines, fiber optic lines, wireless connections, etc. The service provider data network 120 may use a single type of transmission media or a combination of transmission media types. For example, node devices 130 of the service provider data network 120 may transmit and receive data via fiber optic lines through the Internet 160 and transmit data with LANs 140A to 140D using coaxial lines. The service provider data network 120 may include additional node devices connected to different numbers and / or configurations of LANs. Additionally, the service provider data network 120 may include one or more headend systems (not shown) configured to distribute services to node devices, monitor the performance of the service provider data network 120, couple the service provider data network 120 to resources outside the network (e.g., the Internet 160), or manage customer access and billing for services provided by the service provider data network 120.

[0023] The node device 130 of the service provider data network 120 represents a system configured to distribute communications to LANs operating at customer premises within the service provider data network 120. The node device 130 may include hardware and software components configured to translate communication protocols received over one type of transmission medium (e.g., fiber optic line) and distributed over a second type of transmission medium (e.g., coaxial line), and to make routing decisions to direct communications to their desired destination.

[0024] like Figure 1As depicted, LAN 140A includes a gateway device 142, a router 144, and a set-top box (STB) 146. In short, the gateway device 142 is configured to communicatively connect LAN 140A to the service provider data network 120 and facilitate communication between the two networks. The gateway device 142 may be implemented as a Passive Optical Network (PON) modem, a Cable Data Service Interface Specification (DOCSIS) wired modem, a PON Optical Line Terminal (OLT) device, a DOCSIS Remote Physical Layer (RPD) device, or a Remote Physical Layer and MAC Layer (RMD) device, etc. The router 144 is configured to route communications to their desired destination within LAN 140A. The connection between the router 144 and other components of LAN 140A may be wired (e.g., Ethernet) and / or wireless (using the IEEE 802.11 standard WiFi). STB 146 represents an appliance configured to facilitate access to and enjoyment of services provided by the service provider data network 120. For example, video content can be received, decoded, and provided for display on a display device (e.g., television 148). Although gateway device 142, router 144, and STB 146 are depicted as three distinct devices, some or all of the functionality provided by these devices can be combined into a single device, such as a router-gateway device. Aspects of gateway device 142, router 144, and STB 146 are described in more detail below.

[0025] Figure 1This section describes several connected devices communicating with router 144. These connected devices include smartphone 150, thermostat 152, and camera 154. Smartphone 150 represents a connected user device through which a user can access and enjoy services provided by the service provider's data network 120. Services may include data / Internet access, video content, games, etc. This subject matter is not limited to smartphones and may relate to other types of electronic user devices, such as laptops and tablets. Thermostat 152 and camera 154 represent examples of connected Internet of Things (IoT) devices configured to collect data and exchange data with other devices and systems to provide information and functionality to a user. IoT devices can use various sensors (e.g., thermometers, motion sensors, camera arrays, microphones, etc.) to capture data, which can be processed using ML algorithms (e.g., neural network algorithms) to provide functionality to the user (e.g., image detection, voice recognition, efficient thermostat scheduling, etc.). IoT devices typically have limited processing power, and therefore the captured data can be transmitted / streamed from the IoT device to another device or system for further processing (e.g., ML processing). As described herein, the techniques of this subject provide a more efficient method for processing captured data without having to send it to cloud computing resources via the Internet. These techniques are not limited to these types of IoT devices and may relate to other types of IoT devices, such as smart appliances, weather stations, alarm systems, etc.

[0026] LANs 140B to 140D can be implemented using the same or similar components described above for LAN 140A. LANs 140B to 140D can use the same number and type of components as LAN 140A, or can use a different number and / or type of components. For example... Figure 1 The LANs 140B to 140D depicted are represented by images of a building. This subject matter is not limited to residential users and may include LANs installed in multi-family residential areas (e.g., apartment buildings) and commercial premises. Furthermore, any of the LANs 140A to 140D can be implemented as a physical network or a virtualized network. A virtualized network may be based on a virtual LAN (VLAN) identified by a unique VLAN tag, where devices on the VLAN are logically connected to the physical LAN but can share the physical resources of the physical LAN with one or more other VLANs, while communication within the VLAN is isolated from communication in other VLANs.

[0027] Figure 1In this context, cloud controller 180 refers to a server or group of servers configured to receive information from components of service provider data network 120 and process that information for the operators of service provider data network 120. The information may include communications with reports from ML applications running on service provider data network 120. Cloud controller 180 can expose the information to operators for review and decision-making. Cloud controller 180 can also be configured to initiate actions within service provider data network 120 to resolve problems / conditions indicated in the information.

[0028] Figure 2 Examples of routing devices according to aspects of the subject matter are illustrated. However, not all depicted components are necessary, and one or more embodiments may include additional components not shown in the figures. Changes may be made to the arrangement and type of components without departing from the spirit or scope of the claims set forth herein. Unless expressly stated otherwise, the connections or couplings (including electrical and communication connections and couplings) between the depicted or described components are not limited to direct connections or direct couplings, and may be implemented using one or more intermediate components.

[0029] like Figure 2 As depicted herein, router 200 includes processor 210, memory 220, wired interface 230, and wireless interface 240. Processor 210 (e.g., processing circuitry) may include suitable logic, circuitry, and / or code to implement data processing and / or control the operation of router 200. The functionality and operation of router 210 described herein may be implemented using software / firmware (e.g., instructions, code, subroutines, etc.) loaded and executed by processor 210 to provide operation. In this regard, processor 210 may be configured to provide control signals to various other components of router 200. Processor 210 may also control data transfer between components within router 200 and between router 200 and other devices or systems external to router 200. Processor 210 may be implemented using circuitry such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gating logic, discrete hardware components, or any other suitable means.

[0030] Memory 220 may contain suitable logic, circuitry, and / or code, enabling the storage of various types of data and information, such as received data, generated data, code, and / or configuration information. Memory 220 may include, for example, random access memory (RAM), read-only memory (ROM), flash memory, etc. Memory 220 may contain various types of memory, such as volatile memory and non-volatile memory. Figure 2As described herein, memory 220 contains operating system 250, router functionality 260, and ML application client 270, which will be described in further detail below.

[0031] According to aspects of the subject matter, operating system 250 includes a computer program having one or more sequences of instructions or code and associated data and settings. For example, when the processor 210 executes the instructions or code, one or more processes are initiated to manage the resources and operation of router 200 to implement the processes described herein.

[0032] According to aspects of the subject matter, router function 260 includes one or more computer programs having one or more sequences of instructions or codes and associated data and settings. For example, when the instructions or code are executed by processor 210, one or more processes may be initiated to perform routing operations for data communications within a network (e.g., LAN 140A as described herein). Router function 260 may be configured to reference and maintain one or more routing tables or other information for controlling the routing of data communications within the network. Router function 260 may also be configured to respond to and mitigate data traffic conflicts and congestion with the network.

[0033] According to aspects of the subject matter, the ML application client 270 includes one or more computer programs having one or more sequences of instructions or codes and associated data and settings. For example, when the instructions or code are executed by the processor 210, one or more processes may be initiated to monitor one or more devices originating from or connected to the router 200 (e.g., Figure 1 The data streams from the smartphone 150, thermostat 152, and camera 154 are included. The ML application client 270 may include application functions for monitoring the data streams, identifying data types and / or content of interest within the data streams, capturing data from the data streams associated with the identified data types and / or content, generating requests for one or more ML operations to be performed on the captured data, and transmitting the requests and the captured data to an ML-capable device. The ML application client 270 can be used to monitor all data streams transmitted through the router 200, or may be limited to specific types of data streams, or data streams associated with specific connected devices. In this example, the router 200 represents the ML application client device.

[0034] According to aspects of this subject matter, wired interface 230 and wireless interface 240 may include suitable circuitry, logic, and / or code for implementing data packet communication with router 200. Wired interface 230 and wireless interface 240 may also include structural elements to facilitate physical coupling between router 200 and transmission media to provide for the transmission of data packet-encoded electrical signals over the transmission media. This subject matter is not limited to any particular network protocol and / or configuration, and wired protocols (e.g., Ethernet) and / or wireless protocols (e.g., WiFi (IEEE 802.11 standard)) can be used. Each of wired interface 230 and wireless interface 240 may include a single interface through which all data packet communication is performed, and multiple interfaces of the same type for facilitating communication with different (e.g., multiple Ethernet ports, multiple WiFi connections, etc.).

[0035] Figure 3 Examples of gateway devices according to aspects of the subject matter are illustrated. However, not all depicted components are necessary, and one or more embodiments may include additional components not shown in the figures. Changes may be made to the arrangement and type of components without departing from the spirit or scope of the claims set forth herein. Unless expressly stated otherwise, the connections or couplings (including electrical and communication connections and couplings) between the depicted or described components are not limited to direct connections or direct couplings and may be implemented using one or more intermediary components.

[0036] like Figure 3 As depicted herein, gateway device 300 includes processor 310, memory 320, wired interface 330, and wireless interface 340. Processor 310 (e.g., processing circuitry) may include suitable logic, circuitry, and / or code to implement data processing and / or control the operation of gateway device 300. The functionality and operation of gateway device 300 described herein may be implemented using software / firmware (e.g., instructions, code, subroutines, etc.) loaded and executed by processor 310 to provide operation. In this regard, processor 310 may be configured to provide control signals to various other components of gateway device 300. Processor 310 may also control data transfer between components within gateway device 300 and between gateway device 300 and other devices or systems external to gateway device 300. Processor 310 may be implemented using circuitry such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic, discrete hardware components, or any other suitable means.

[0037] Memory 320 may contain suitable logic, circuitry, and / or code, enabling the storage of various types of data and information, such as received data, generated data, code, and / or configuration information. Memory 320 may include, for example, random access memory (RAM), read-only memory (ROM), flash memory, etc. Memory 320 may include various types of memory, such as volatile memory and non-volatile memory. Figure 3 As described herein, memory 320 contains operating system 350, gateway function 360, and ML agent client 370, which will be described in further detail below.

[0038] According to aspects of the subject matter, operating system 350 includes a computer program having one or more sequences of instructions or code and associated data and settings. For example, when the processor 310 executes the instructions or code, one or more processes are initiated to manage the resources and operations of gateway device 300 to implement the processes described herein.

[0039] According to aspects of the subject matter, gateway function 360 includes one or more computer programs having one or more instruction sequences or codes and associated data and settings. For example, when the instructions or code are executed by processor 310, one or more processes may be initiated to perform gateway operations for two different networks (e.g., Figure 1 The depicted data communication occurs between the service provider data network 120 and LAN 140A. Gateway operations may include formatting, translation, signaling, etc., to facilitate the flow of data packets from one type of network to another. Gateway operations may also include security features to protect the content of data packets and restrict or prohibit the transmission of specified types of data and data transmission from specified locations.

[0040] According to aspects of the subject matter, the ML agent client 370 includes one or more computer programs having one or more sequences of instructions or codes and associated data and settings. For example, when the instructions or code are executed by the processor 310, one or more processes may be initiated to handle requests for ML processing to be performed and to communicate the results of the requested ML processing. ML processing generally refers to performing one or more ML operations on a set of data to generate a set of results. ML operations may include inference operations, in which one or more ML models or algorithms are applied to a set of data to generate predictions or results. Results may contain any type of structured or unstructured data (e.g., text, images, numerical values, etc.). ML operations may further include training operations, in which an ML model or algorithm is applied to one or more sets of training data or labeled data to determine and refine the parameters of the ML model or algorithm. The parameters of the ML model or algorithm can be determined and refined by iteratively comparing the results of the ML model or algorithm with the desired results indicated in the labeled data and updating the parameters based on the comparisons during iteration.

[0041] ML agent client 370 can be configured to communicate with ML application clients regarding requested ML processing and data to be applied to ML operations associated with the requested ML processing. ML agent client 370 can be further configured to communicate with one or more ML agent servers regarding requested ML processing, data to be applied to ML operations associated with the requested ML processing, and the results of the ML processing. For example, ML agent client 370 can receive an ML application request from an ML application client to perform an ML operation on a set of captured data. ML agent client 370 can process the ML application request to identify the ML operation to be performed and any preprocessing requirements associated with the ML operation. In this context, preprocessing refers to any type of modification and / or organization performed on a set of data before an ML model or algorithm is applied to it. ML agent client 370 can preprocess a set of captured data according to the identified preprocessing requirements (e.g., setting / translating data size, data type, data format, arrangement of data content, etc.).

[0042] To perform the requested ML operation, the ML agent client 370 may be configured to generate an ML agent request that identifies the requested ML operation and contains a preprocessed set of data or the retrieval location of said data, and send the ML agent request to the ML agent server to execute the ML operation. After sending the ML agent request, the ML agent client 370 may be configured to receive an ML agent response from the ML agent server, which contains the results of the ML processing and / or messages regarding the execution of the ML processing. The ML agent client 370 may be further configured to generate a report containing the results of the ML processing or indications of those results, to (e.g., via...) Figure 1 The cloud controller 180 in the ML agent client 370 sends the data to the service provider's data network. For example, when generating a report, the ML agent client 370 may be configured to transform and / or format the results of the ML operation to a specification provided by the service provider, and apply security measures when generating and transmitting the report outside the LAN. Security measures may include, but are not limited to, various encryption techniques for securely transmitting the report outside the LAN and applying measures such as differential privacy algorithms to the data results to address individual privacy concerns. A differential privacy algorithm can refer to a mathematical algorithm applied when generating the results of an ML operation to limit or prevent the ability to identify individual elements of the input dataset based on the output results without negatively impacting the output results. In the foregoing example, the gateway device 300 represents the ML agent client device.

[0043] According to aspects of this subject matter, the wired interface 330 and cable interface 340 may include suitable circuitry, logic, and / or code for implementing data packet communication with the gateway device 300. The wired interface 330 and cable interface 340 may also include structural elements to facilitate physical coupling between the gateway device 300 and the transmission medium to provide for the transmission of data packet-encoded electrical signals over the transmission medium. This subject matter is not limited to any particular network protocol and / or configuration, and protocols such as Ethernet based on the wired interface 330 and DOCSIS based on the cable interface 340 may be used. Each of the wired interface 330 and cable interface 340 may include a single interface through which all data packet communication is performed, and multiple interfaces of the same type for facilitating communication with different (e.g., multiple Ethernet ports, multiple coaxial connections, etc.). This subject matter is not limited to these types of interfaces and other types of interfaces may be used for implementation, for example, optical transmission lines (e.g., PON devices) and / or wireless data transmission (e.g., WiFi, IEEE 802.11 standard).

[0044] Figure 4Examples of multifunctional electrical appliances (e.g., set-top boxes) according to aspects of the subject matter are illustrated. However, not all depicted components are essential, and one or more embodiments may include additional components not shown in the figures. Changes may be made to the arrangement and type of components without departing from the spirit or scope of the claims set forth herein. Unless expressly stated otherwise, the connections or couplings (including electrical and communication connections and couplings) between the depicted or described components are not limited to direct connections or direct couplings and may be implemented using one or more intermediate components.

[0045] like Figure 4 As depicted herein, a set-top box (STB) 400 includes a processor 410, a memory 420, an audio / video (A / V) interface 430, and a network interface 440. The processor 410 (e.g., a processing circuitry) may include suitable logic, circuitry, and / or code to implement data processing and / or control the operation of the STB 400. The functions and operations of the STB 400 described herein may be implemented using software / firmware (e.g., instructions, code, subroutines, etc.) loaded and executed by the processor 410 to provide operation. In this regard, the processor 410 may be configured to provide control signals to various other components of the STB 400. The processor 410 may also control data transfer between components within the STB 400 and between the STB 400 and other devices or systems external to the STB 400. The processor 410 may be implemented using circuitry such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gating logic, discrete hardware components, or any other suitable means.

[0046] Memory 420 may contain suitable logic, circuitry, and / or code, enabling the storage of various types of data and information, such as received data, generated data, code, and / or configuration information. Memory 420 may include, for example, random access memory (RAM), read-only memory (ROM), flash memory, etc. Memory 420 may contain various types of memory, such as volatile memory and non-volatile memory. Figure 4 As described herein, memory 420 includes operating system 460, STB function 470, ML agent server 480, and power manager 495, which will be described in further detail below.

[0047] According to aspects of the subject matter, the operating system 460 includes a computer program having one or more sequences of instructions or code and associated data and settings. For example, when the processor 410 executes the instructions or code, one or more processes are initiated to manage the resources and operations of the STB 400 to implement the processes described herein.

[0048] According to aspects of the subject matter, STB function 470 includes one or more computer programs having one or more instruction sequences or codes and associated data and settings. For example, when the processor 410 executes the instructions or code, one or more processes may be initiated to perform STB operations for accessing and utilizing services (e.g., video content, data, telephone, games, etc.) provided by a service provider's data network. STB operations may include communicating with a user via one or more user interfaces to select / control services accessed via STB 400 and receiving user feedback / input regarding the services. STB operations may also include decoding content using one or more audio / video CODECs (e.g., H.264, H.265, H.266, VP9, ​​AV1, etc. for video content and AC3, AAC, He-AAC, MP3, WAV, etc. for audio content) for presentation by STB 400.

[0049] According to aspects of the subject matter, the ML agent server 480 includes one or more computer programs having one or more sequences of instructions or code and associated data and settings. For example, when the instructions or code are executed by the processor 410, one or more processes may be initiated to perform agent functions, including, but not limited to, processing ML agent requests received from ML agent clients, performing ML operations indicated in the ML agent requests on input data, and conveying the results of the ML operations back to the ML agent client in ML agent response communications. In this example, STB 400 represents an ML agent server apparatus.

[0050] ML operations may involve applying a set of input data to a trained ML model to perform inference on the input data. The inference result may identify patterns in the input data and / or make predictions about the input data. For example, an ML model may be trained to identify and / or track objects in image data captured by a camera. Another ML model may be trained to predict future network performance or usage from a dataset capturing current or previous network traffic and conditions. The techniques in this subject matter are not limited to any particular type of ML model.

[0051] ML models may involve taking data as input to a mathematical algorithm that generates inference results as output. An ML model can be defined using a set of parameters that identify the values ​​of variables used in the mathematical algorithm (e.g., neural network model operations and parameters), and can specify the type and format of the input data. An ML agent server 480 may be able to perform various types of inference operations using different sets of model parameters. Figure 4The ML model 490 depicted represents a parameter set library stored in memory 420 for use with ML models. Based on an ML agent request received from an ML agent client, the ML agent server 480 is operable to select and retrieve a parameter set corresponding to the ML operation or inference indicated in the ML agent request from the ML model 490. The parameter set is used to invoke and configure the ML processing engine 450 to perform inference.

[0052] According to aspects of the subject matter, the ML processing engine 450 (e.g., a neural processing unit) may include suitable logic, circuitry, and / or code configurable to execute ML models to perform inference on a set of input data. The ML processing engine 450 may operate as a hardware accelerator, wherein the logic and circuitry of the ML processing engine 450 may be configured to implement different ML models using associated parameter sets. The ML processing engine 450 may be implemented using circuitry from, for example, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gating logic, discrete hardware components, or any other suitable means.

[0053] The proxy functionality provided by the ML proxy server 480 may further include formatting the inference results generated by the ML processing engine 450 and / or conveying the inference results back to the requesting ML proxy client in the proxy response.

[0054] According to aspects of the subject matter, power manager 495 includes one or more computer programs having one or more instruction sequences or codes and associated data and settings. For example, when the instructions or code are executed by processor 410, one or more processes may be initiated to manage the operating mode and associated power consumption of STB 400. As described above, STB 400 can be used for both STB functionality and STB function 470 to access and enjoy services provided by the operator and to perform agent functions using ML agent server 480. For example, power manager 495 may include a process for monitoring both STB operation and ML agent operation within STB 400 to determine the appropriate operating mode of STB 400 (e.g., sleep mode). Even in the absence of STB functionality during operation, STB 400 can provide ML agent functionality to perform requested inference. Therefore, power manager 495 may be configured to wait for a period of time before allowing STB 400 to enter sleep mode, during which time both STB functionality and ML agent functionality are inactive.

[0055] According to aspects of this subject matter, the audio / video (A / V) interface 430 may include suitable circuitry, logic, and / or code for presenting content and various communications to a user via one or more peripheral devices (e.g., televisions and speakers) connected to the STB 400 through the A / V interface 430. In addition to content and communication, the peripheral devices connected to the STB 400 can be used to facilitate interaction with one or more user interfaces. For example, connecting a keyboard, control pad, pointing device, etc., can facilitate the user's selection of user interface items presented to be displayed by the STB function 470. One or more microphones may be connected to the STB 400 and used to capture audio signals, including but not limited to user voice as user interface input, test signals for calibration operations, etc. The A / V interface 430 may also include structural elements to facilitate physical coupling between the STB 400 and the transmission medium used for transmitting and receiving signals (e.g., audio signals, video signals, etc.) from and via the connected peripheral devices. This subject matter is not limited to any particular protocol or interface and can be used for both analog and digital signal transmission.

[0056] As indicated above, one or more microphones may be connected to the STB 400 to facilitate the capture of audio signals (e.g., user voice). ML capabilities available in the STB 400 may be accessed to improve user interaction with the STB 400. For example, the ML model 490 may contain a set of parameters for performing speech-to-text operations, text-to-speech operations, image generation, video generation, etc. These parameter sets may be used to invoke and configure the ML processing engine 450 to process user voice captured by the microphone to generate text for display to the user and / or input to user interface elements, to process text generated by components of the STB 400 or text received by the STB 400 from another device, to present audio to the user via one or more speakers connected to the STB 400, etc.

[0057] According to aspects of this subject matter, network interface 440 may include suitable circuitry, logic, and / or code for implementing data communication with STB 400. Network interface 440 may also include structural elements to facilitate physical coupling between STB 400 and transmission media to provide data-encoded electrical signals for transmission over the transmission media. This subject matter is not limited to any particular network protocol and / or configuration, and may use, for example, Ethernet based on a wired interface and WiFi based on a wireless interface (e.g., the IEEE 802.11 protocol). Network interface 440 may include a single interface through which all data communication is performed, multiple interfaces of the same type (e.g., multiple Ethernet ports, multiple WiFi connections, etc.) to facilitate communication with different types of network interfaces, or combinations of different types of network interfaces.

[0058] Figure 5This is a flowchart illustrating the operation of a machine learning agent system based on aspects of the subject matter. For illustrative purposes, the blocks of the illustrated process may be described herein as occurring sequentially or linearly. However, two or more blocks of the illustrated process may be executed in parallel. Furthermore, Figure 5 The boxes depicted may be executed in a different order than those shown, and the process may not execute one or more of the boxes described and / or may include one or more additional boxes.

[0059] Figure 5 The process described herein includes operations performed by an ML application client, an ML agent client, and an ML agent server (such as the instances described above) and communication between them to implement the functions and operations described herein. According to an aspect of the subject matter, the process includes the ML application client monitoring data transmitted via a connected device on the LAN (box 500), processing the monitored data using ML application functions to generate an ML application request to perform inference operations on data captured from the monitored data (box 505), and sending the ML application request to the ML agent client on the LAN (box 510).

[0060] Figure 6 This is to explain the technical aspects of this topic. Figure 5 The diagrams in blocks 500, 505, and 510 illustrate the data flow associated with the operation of the ML application client. However, not all depicted components are essential, and one or more embodiments may include additional components not shown in the figures. Changes in the arrangement and type of components may be made without departing from the spirit or scope of the claims set forth herein. Unless expressly stated otherwise, the connections between the depicted or described components are not limited to direct connections and may be implemented using one or more intermediary components.

[0061] like Figure 6 As described, the WiFi router 600 (e.g., an ML application client device) includes a WiFi interface 610, an ML application client 620, a router 630, and an Ethernet interface 640. Figure 6 The document also describes an IoT device 650 communicating with a WiFi interface 610, and a cable gateway 660 and an ML agent client 670 communicating with an Ethernet interface 640. Implementations and operations of these components are provided above and will not be repeated here.

[0062] According to aspects of the technology in this subject matter, the IoT device 650 can send and / or receive data from... Figure 6 The data streams represent one or more data streams as defined by the IoT device IP service. The data streams may originate from the IoT device 650, or from a device containing... Figure 6 The components described herein are different entities outside the LAN. For example... Figure 6 As indicated, IoT device IP services are received from IoT device 650 via WiFi interface 610 or from cable gateway 660 via Ethernet interface 640. Regardless of the origin of the IoT device IP services, the data stream containing the data is processed by ML application client 620 to identify the data type, pattern, content, etc., in the data stream that triggers inference operations on the data captured from the data stream, based on the functions provided by ML application client 620.

[0063] When the ML application client 620 needs to perform an inference operation on data captured from the monitored data, an ML application request containing an indication of the type of inference operation requested and the captured data (or the location of the captured data) is generated by the ML application client 620 and sent via router 630 and Ethernet interface 640. Figure 6 The ML agent client 670 is implemented on the cable gateway 660 in the middle.

[0064] Return to Figure 5 The process continues by the ML agent client processing the ML application request received from the ML application client to determine the ML processing requirements associated with the requested inference operation (box 515). The ML processing requirements may include an identifier of the ML model to be used for the requested inference processing, and any data formatting specifications for the input data. Based on the determined ML processing requirements, the ML agent client generates an ML agent request (box 520) and sends it to the ML agent server on the LAN (box 525). The ML agent request may include an identifier of the ML model to be used as input to the ML model and formatted data, or the location of formatted data.

[0065] According to the technical aspects of this topic, ML agent clients and ML agent servers can perform initialization steps ( Figure 5 (Not described in the text) to perform mutual authentication between the ML agent client and the ML agent server. Mutual authentication is not limited to any particular type or number of authentication algorithms, and for example, public-key algorithms based on post-quantum cryptography can be used. Mutual authentication can be performed when the ML agent client and ML agent server are started, after the ML agent client receives a request from the ML application, or at any other time before the ML agent client sends an ML agent request to the ML agent server.

[0066] Figure 7 This is to explain the technical aspects of this topic. Figure 5The diagrams in boxes 515, 520, and 525 illustrate the data flow associated with the operation of the ML agent client. However, not all depicted components are essential, and one or more embodiments may include additional components not shown in the figures. Changes in the arrangement and type of components may be made without departing from the spirit or scope of the claims set forth herein. Unless expressly stated otherwise, the connections between the depicted or described components are not limited to direct connections and may be implemented using one or more intermediary components.

[0067] like Figure 7 As described, the cable gateway 700 (e.g., an ML agent client device) includes an ML agent client 710, an Ethernet interface 720, and a cable interface 730. Figure 7 The document also describes a WiFi router 740, an ML application client 750, an ML proxy server 760 communicating with an Ethernet interface 720, and a cloud controller 770 and an Internet 780 communicating with a cable interface 730. Implementations and operations of these components are provided above and will not be repeated here.

[0068] According to an aspect of the subject matter, an ML agent client 710 receives an ML application request from an ML application client 750 on a WiFi router 740 via an Ethernet interface 720. As described in the example above, the received ML application request is processed by the ML agent client 710 to determine ML processing requirements for generating an ML agent request. The ML agent request is sent to an ML agent server 760 on a set-top box (STB) via the Ethernet interface 720. The STB may have an Ethernet connection to a cable gateway 700 that does not pass through the WiFi router 740, thereby allowing the ML agent request to be sent to the ML agent server 760 bypassing the WiFi router 740. Alternatively, the ML agent request may be sent to the ML agent server 760 via the WiFi router 740.

[0069] Return to Figure 5 The process continues by the ML proxy server initiating an ML model and generating an inference request based on the ML proxy request received from the ML proxy client (box 530). The initiated ML model can perform inference based on the ML inference request and output the inference results used by the ML proxy server to generate the ML proxy response (box 535). The ML proxy server sends the ML proxy response to the ML proxy client that made the request (box 540).

[0070] Figure 8 This is to explain the technical aspects of this topic. Figure 5The diagram illustrates the data flow associated with the operation of the ML agent server as described in boxes 515, 520, and 525. However, not all depicted components are essential, and one or more embodiments may include additional components not shown in the figures. Changes in the arrangement and type of components may be made without departing from the spirit or scope of the claims set forth herein. Unless expressly stated otherwise, the connections between the depicted or described components are not limited to direct connections and may be implemented using one or more intermediary components.

[0071] like Figure 8 As described, the STB 800 (e.g., an ML proxy server device) includes a WiFi / Ethernet interface 810, an ML proxy server 820, an ML driver / API 830, an ML processing engine 840, and an ML model 850. Figure 8 The document also depicts an ML agent client 860 communicating with a WiFi / Ethernet interface 810 on a cable gateway and an A / V peripheral device 870 communicating with an ML agent server 820. Implementations and operations of these components are provided above and will not be repeated here.

[0072] According to an aspect of the subject matter, an ML agent server 820 receives an ML agent request from an ML agent client 860 via a WiFi / Ethernet interface 810. The ML agent server 820 generates an ML inference request based on the received ML agent request and starts an ML model on an ML processing engine 840 using ML model parameters retrieved from an ML model 850. Input data for the ML inference request is provided to the ML processing engine 840 via an ML driver / API 830, and inference operations on the input data are performed by the ML processing engine 840. An ML inference response containing the inference results is provided by the ML processing engine 840 to the ML agent server 820 via the ML driver / API 830. The ML agent server 820 generates an ML agent response containing the inference results and sends the ML agent response to the ML agent client 860 via the WiFi / Ethernet interface 810.

[0073] Return to Figure 5 The ML agent client generates an ML application report (box 545) based on the received ML agent response and sends the generated ML application report to a controller outside the LAN (box 550). As mentioned above, the ML agent client can use various technologies and mechanisms to address privacy concerns related to sending inference results outside the LAN. Figure 7 As described, the ML agent client 710 sends ML application reports to the cloud controller 770 via the cable interface 730.

[0074] As indicated in the examples above, the various components of the machine learning agent system described herein can be implemented in different devices on a LAN. For example, the ML application client can be implemented in a WiFi router, the ML agent client in a cable gateway, and the ML agent server in an STB. However, the subject matter is not limited to these configurations. For example, the ML application client can be implemented directly in a connected device on the LAN, or alternatively, in a cable gateway device. The ML agent server and ML processing engine can be implemented in a cable gateway device instead of an STB. Additionally, the ML agent client can be implemented in a node device within a service provider's data network. Using this last configuration allows the ML agent client to send ML agent requests to ML agent servers located in different LANs within the service provider's data network.

[0075] Figure 9 This is a block diagram illustrating the data flow associated with the operation of the ML application client and ML agent client in alternative configurations according to aspects of the subject matter. However, not all depicted components are necessary, and one or more embodiments may include additional components not shown in the diagram. Changes may be made to the arrangement and type of components without departing from the spirit or scope of the claims set forth herein. Unless expressly stated otherwise, the connections between the depicted or described components are not limited to direct connections and may be implemented using one or more intermediary components.

[0076] According to the technical aspects of this subject matter, the cable gateway 900 includes an ML application client 905, an ML agent client 910, a cable interface 915, and an Ethernet interface 920. A WiFi router 925 and its STB implementing an ML agent server 930 communicate with the cable gateway 900 via the Ethernet interface 920. A cloud controller 935 and an Internet 940 communicate with the cable gateway 900 via the cable interface 915. Figure 9 The example described illustrates a configuration where the ML application client 915 is implemented on the cable gateway 900 instead of the WiFi router 915. Therefore, as data passes through the cable gateway 900, IoT device IP traffic is monitored and captured by the ML application client 915, and ML application requests are generated by the ML application client 915 and sent to the ML agent client 915 within the cable gateway 900. In addition to the aforementioned changes in data flow, Figure 9 Other data streams described above are as follows: Figure 6 and 7 It operates as described. In this example, the ML application client and the ML agent client are implemented on a single electronic device (e.g., cable gateway 900).

[0077] Figure 10This is a block diagram illustrating the data flow associated with the operation of the ML agent client and ML agent server in an alternative configuration according to aspects of the subject matter. However, not all depicted components are necessary, and one or more embodiments may include additional components not shown in the diagram. Changes may be made to the arrangement and type of components without departing from the spirit or scope of the claims set forth herein. Unless expressly stated otherwise, the connections between the depicted or described components are not limited to direct connections and may be implemented using one or more intermediary components.

[0078] According to aspects of the subject matter, the cable gateway 1000 includes an ML agent client / server 1005, an ML driver / API 1010, an ML processing engine 1015, an ML model 1020, a cable interface 1025, and an Ethernet interface 1030. A WiFi router 1035 and an ML application client 1040 implemented on the WiFi router 1035 communicate with the cable gateway 1000 via the Ethernet interface 1030. A cloud controller 1045 and the Internet 1050 communicate with the cable gateway 1000 via the cable interface 1025. Figure 10 The example described illustrates a configuration where the ML agent server 1005, ML processing engine 1015, ML driver / API 1010, and ML model 1020 are implemented on the cable gateway 1000 instead of the STB. Therefore, the generation and delivery of ML inference requests and responses are performed within the cable gateway 1000. Furthermore, the retrieval of model parameters from the ML model 102 and the configuration and operation of the ML processing engine 1015 are performed within the cable gateway 1000. In addition to the aforementioned changes in data flow, Figure 10 Other data streams described above are as follows: Figure 7 and 8 As described. In this example, the ML agent client and ML agent server are implemented on a single electronic device (e.g., cable gateway 1000).

[0079] Figure 11 This is a block diagram illustrating the data flow associated with the operation of the ML application client, ML agent client, and ML agent server in alternative configurations according to aspects of the subject matter. However, not all depicted components are essential, and one or more embodiments may include additional components not shown in the diagram. Changes may be made to the arrangement and type of components without departing from the spirit or scope of the claims set forth herein. Unless expressly stated otherwise, the connections between the depicted or described components are not limited to direct connections and may be implemented using one or more intermediary components.

[0080] According to aspects of the subject matter technology, cable gateway 1100 includes an ML application client 1105, an ML agent client / server 1110, an ML driver / API 1115, an ML processing engine 1120, an ML model 1125, a cable interface 1130, and an Ethernet interface 1135. WiFi router 1140 communicates with cable gateway 1100 via Ethernet interface 1135. Cloud controller 1145 and Internet 1150 communicate with cable gateway 1100 via cable interface 1130. Figure 11 The example described illustrates a configuration where the ML application client and ML proxy server are implemented on the cable gateway 1100, rather than on the WiFi router 1140 and the STB respectively. Therefore, the change to the data flow is... Figure 9 and 10 The combination of data streams represented in the figure. In this example, the ML application client, ML agent client, and ML agent server are all implemented on a single electronic device (e.g., cable gateway 1100).

[0081] According to an aspect of the present subject matter, an electronic device is provided, comprising: a first network interface configured to transmit and receive communications on a first network; a second network interface configured to transmit and receive communications on a second network different from the first network; a computer-readable storage medium storing one or more instruction sequences; and a processing circuitry system configured to execute the one or more instruction sequences to perform an operation. The operation may include receiving an application request from an ML application client device on the first network, wherein the application request includes application data captured by the ML application client device from one or more data streams transmitted via the first network and the second network to a connected device on the first network; sending a proxy request to an ML proxy server on the first network, wherein the proxy request includes the application data and an indicated inference operation to be performed on the application data; receiving a proxy response from the ML proxy server, wherein the proxy response includes the result of the inference operation performed by the ML proxy server on the application data; and sending a report based on the proxy response to a controller outside the first network via the second network interface.

[0082] The operation may further include preprocessing the application data based on parameters associated with the indicated inference operation, wherein the proxy request sent to the ML proxy server includes the preprocessed application data. The operation may further include processing the result of the inference operation received in the proxy response, wherein the report sent to the controller includes the processed result. Processing the result of the inference operation may include applying a differential privacy algorithm to the inference result.

[0083] The ML proxy server device may be configured to send user notifications based on the results of the inference operation for audio or visual presentation to a user via one or more peripheral devices communicating with the ML proxy server device, and to receive user responses to the presented user notifications, wherein the user responses are captured via a user interface. The user interface includes a microphone and a natural language processor executed using a neural processing unit on the ML proxy server device.

[0084] The operation may further include performing mutual authentication with the ML proxy server device. The ML application client device and the electronic device are a single device, and the operation may further include monitoring the application data with the one or more data streams transmitted by the connected device on the first network; and capturing the application data.

[0085] The first network may be a LAN operating at a first location, and the second network may be a service provider data network providing communication for multiple LANs operating at multiple different locations. The first network may be a virtualized local area network.

[0086] According to an aspect of the present subject matter, a method is provided comprising: receiving an application request from an ML application client device on a first network, wherein the application request includes application data captured by the ML application client device from one or more data streams communicated with a connected device on the first network and an indicated inference operation to be performed on the application data; preprocessing the application data based on parameters associated with the indicated inference operation; sending a proxy request to an ML proxy server on the first network, wherein the proxy request includes the preprocessed application data and the indicated inference operation to be performed on the preprocessed application data; receiving a proxy response from the ML proxy server, wherein the proxy response includes a result of the inference operation performed by the ML proxy server on the application data; and sending a report based on the proxy response to a controller on a second network outside the first network.

[0087] The method may further include processing the result of the inference operation received in the agent response, wherein the report sent to the controller includes the processed result. Processing the result of the inference operation may include applying a differential privacy algorithm to the inference result.

[0088] The method may further include sending a user notification based on the result of the inference operation to a network device on the first network via one or more peripheral devices communicating with the network device for audio or visual presentation to a user; and receiving a user response to the presented user notification from the network device via a user interface.

[0089] The method may further include monitoring the one or more data streams transmitted by the connected device on the first network for application data; and capturing the application data associated with the ML application client.

[0090] According to an aspect of the present subject matter, a system is provided comprising: an ML application client configured to perform on a first device on a first network to: monitor one or more data streams transmitted via a connected device on the first network for application data associated with the ML application client; and capture the application data associated with the ML application client; an ML proxy server configured to perform on a second device on the first network to: initiate and perform ML inference operations; and an ML proxy client configured to perform on a third device on the first network to: receive an application request from the ML application client, wherein the application request includes application data captured by the ML application client from one or more data streams transmitted via the connected device; send a proxy request to the ML proxy server, wherein the proxy request includes the application data and an indicated inference operation to be performed on the application data; receive a proxy response from the ML proxy server, wherein the proxy response includes the result of the inference operation performed by the ML proxy server on the application data; and send a report based on the proxy response to a controller outside the first network.

[0091] The ML agent client may be further configured to preprocess the application data based on parameters associated with the indicated inference operation, wherein the agent request sent to the ML agent server includes the preprocessed application data. The ML agent client may be further configured to process the result of the inference operation received in the agent response, wherein the report sent to the controller includes the processed result.

[0092] The first device may be the connected device. The first device and the second device may be gateway devices configured to communicatively couple the first network to a second network different from the first network.

[0093] The embodiments within the scope of this disclosure may be implemented in whole or in part using tangible computer-readable storage media (or multiple tangible computer-readable storage media of one or more types) encoding one or more instructions. The tangible computer-readable storage media may also be non-transitory.

[0094] Computer-readable storage media can be any storage medium that can be read, written, or otherwise accessed by a general-purpose or special-purpose computing device, including any processing electronic device and / or processing circuitry system capable of executing instructions. For example, but not limited to, computer-readable media can include any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. Computer-readable media can also include any non-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, flash memory, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, racetrack memory, FJG, and millipede memory.

[0095] Furthermore, the computer-readable storage medium may comprise any non-semiconductor memory, such as optical disc storage, magnetic disk storage, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In some embodiments, the tangible computer-readable storage medium may be directly coupled to a computing device, while in other embodiments, the tangible computer-readable storage medium may be indirectly coupled to a computing device via one or more wired connections, one or more wireless connections, or any combination thereof.

[0096] Instructions can be directly executable or used to develop executable instructions. For example, instructions can be implemented as executable or non-executable machine code, or as instructions in a high-level language that can be compiled to produce executable or non-executable machine code. Furthermore, instructions can also be implemented as data or contain data. Computer executable instructions can also be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As recognized by those skilled in the art, details including, but not limited to, the number, structure, order, and organization of instructions can vary significantly without altering the underlying logic, functionality, processing, and output.

[0097] The foregoing description is provided to enable those skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects presented herein, but should be accorded the full scope consistent with the language of the claims, wherein, unless expressly stated otherwise, an element referred to in the singular is not intended to mean “one and only one,” but rather “one or more.” Unless expressly stated otherwise, the term “some” refers to one or more. Masculine (e.g., his) pronouns include feminine and neuter (e.g., her and its), and vice versa. Titles and subtitles (if any) are provided for convenience only and do not limit the disclosure of this subject matter.

[0098] The predicates “configured to,” “operable to,” and “programmed to” do not imply any specific tangible or intangible modification to the subject, but are intended to be used interchangeably. For example, a processor configured to monitor and control operations or components may also mean that the processor is programmed to monitor and control operations or that the processor is operable to monitor and control operations. Similarly, a processor configured to execute code can be interpreted as a processor programmed to execute code or operable to execute code.

[0099] For example, the phrase "aspect" does not imply that such an aspect is essential to the present subject matter, or that such an aspect applies to all configurations of the present subject matter. Disclosures relating to an aspect may apply to all configurations or one or more configurations. For example, the phrase "aspect" may refer to one or more aspects, and vice versa. For example, the phrase "configuration" does not imply that such a configuration is essential to the present subject matter, or that such a configuration applies to all configurations of the present subject matter. Disclosures relating to configuration may apply to all configurations or one or more configurations. For example, the phrase "configuration" may refer to one or more configurations, and vice versa.

[0100] The term “example” is used in this document to mean “served as an example or illustration.” Any aspect or design described as an “example” in this document is not necessarily to be construed as being better or more advantageous than other aspects or designs.

[0101] All structural and functional equivalents of elements throughout the various aspects described herein, known or subsequently learned by one of ordinary skill in the art, are expressly incorporated herein by reference and are intended to be covered in the claims. Furthermore, nothing disclosed herein is intended to be exclusive to the public, whether or not such disclosure is expressly stated in the claims. No claim element should be interpreted in accordance with 35 USC § 112(f) unless the element is expressly stated using the phrase “component for…” or, in the case of a method claim, the element is stated using the phrase “step for…”. Furthermore, where the terms “comprising,” “having,” or similar terms are used in the description or claims, such terms are intended to be inclusive in a manner similar to the term “comprising,” as interpreted when “comprising” is used as a transition word in a claim.

[0102] Those skilled in the art will understand that the various illustrative blocks, modules, elements, components, methods, and algorithms described herein can be implemented as electronic hardware, computer software, or a combination of both. To illustrate this interchangeability between hardware and software, the various illustrative blocks, modules, elements, components, methods, and algorithms have been described above in general terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. Skilled technicians can implement the described functionality in different ways for each specific application. Various components and blocks can be arranged differently (e.g., in different orders or divided in different ways), all without departing from the scope of the subject matter.

[0103] The predicates “configured to,” “operable to,” and “programmed to” do not imply any specific tangible or intangible modification to the subject, but are intended to be used interchangeably. For example, a processor configured to monitor and control operations or components could also mean a processor programmed to monitor and control operations or an operable processor to monitor and control operations. Similarly, a processor configured to execute code can be interpreted as a processor programmed to execute code or an operable processor to execute code.

Claims

1. An electronic device comprising: A first network interface, configured to send and receive communications on a first network; A second network interface is configured to send and receive communications on a second network different from the first network. A computer-readable storage medium storing one or more sequences of instructions; and A processing circuitry system configured to execute the one or more instruction sequences to perform operations including: The machine learning ML application client device on the first network receives an application request, wherein the application request includes application data captured by the ML application client device from one or more data streams transmitted via the first network and the second network and a connected device on the first network; Send a proxy request to an ML proxy server device on the first network, wherein the proxy request includes the application data and an indicated inference operation to be performed on the application data; Receive a proxy response from the ML proxy server device, wherein the proxy response includes the result of the inference operation performed by the ML proxy server device on the application data; and A report based on the agent's response is sent to a controller outside the first network via the second network interface.

2. The electronic device according to claim 1, wherein the operation further comprises: The application data is preprocessed based on parameters associated with the indicated inference operation. The proxy request sent to the ML proxy server device includes the preprocessed application data.

3. The electronic device according to claim 1, wherein the operation further comprises: Process the result of the inference operation received in the agent response. The report sent to the controller includes the processed results.

4. The electronic device of claim 3, wherein processing the result of the inference operation includes applying a differential privacy algorithm to the inference result.

5. The electronic device of claim 1, wherein the ML proxy server device is configured to: User notifications based on the results of the inference operation are sent to the user via one or more peripheral devices that communicate with the ML agent server device for audio or visual presentation. Receive user responses to the presented user notifications. The user response is captured via the user interface.

6. The electronic device of claim 5, wherein the user interface includes a microphone and a natural language processor executed using a neural processing unit on a network device.

7. The electronic device of claim 1, wherein the operation further comprises: Perform mutual authentication with the ML proxy server device.

8. The electronic device of claim 1, wherein the ML application client device and the electronic device are a single device, and The operation further includes: For application data monitoring and the one or more data streams transmitted via the connected device on the first network; and Capture the application data.

9. The electronic device of claim 1, wherein the first network is a local area network operating at a first location, and the second network is a service provider data network providing communication for multiple local area networks operating at multiple different locations.

10. The electronic device of claim 1, wherein the first network is a virtualized local area network.

11. A method comprising: The application request is received from a machine learning ML application client device on a first network, wherein the application request includes application data captured by the ML application client device from one or more data streams communicated with a connected device on the first network and an indicated inference operation to be performed on the application data. The application data is preprocessed based on parameters associated with the indicated inference operation; Send a proxy request to an ML proxy server device on the first network, wherein the proxy request includes the preprocessed application data and the indicated inference operation to be performed on the preprocessed application data. Receive a proxy response from the ML proxy server device, wherein the proxy response includes the result of the inference operation performed by the ML proxy server device on the application data; and Send a report based on the agent's response to a controller on a second network outside the first network.

12. The method of claim 11, further comprising: Process the result of the inference operation received in the agent response. The report sent to the controller includes the processed results.

13. The method of claim 12, wherein processing the result of the inference operation includes applying a differential privacy algorithm to the inference result.

14. The method of claim 11, further comprising: Perform mutual authentication with the ML proxy server device.

15. The method of claim 11, further comprising: For application data monitoring and the one or more data streams transmitted via the connected device on the first network; and Capture the application data.

16. A system comprising: A machine learning (ML) application client, configured to execute on a first device on a first network, to: Monitoring of application data associated with the ML application client and one or more data streams transmitted via a connected device on the first network; and Capture the application data associated with the ML application client; An ML proxy server, configured to perform the following on a second device on the first network: Initiate and execute ML inference operations; and An ML agent client, configured to execute on a third device on the first network, to: Receive an application request from the ML application client, wherein the application request includes application data captured by the ML application client from one or more data streams communicated with the connected device; Send a proxy request to the ML proxy server, wherein the proxy request includes the application data and the indicated inference operation to be performed on the application data; Receive a proxy response from the ML proxy server, wherein the proxy response includes the result of the inference operation performed by the ML proxy server on the application data; and Send a report based on the agent's response to a controller outside the first network.

17. The system of claim 16, wherein the ML agent client is further configured to: The application data is preprocessed based on parameters associated with the indicated inference operation. The proxy request sent to the ML proxy server includes the preprocessed application data.

18. The system of claim 16, wherein the ML agent client is further configured to: Process the result of the inference operation received in the agent response. The report sent to the controller includes the processed results.

19. The system of claim 16, wherein the first device is the connected device.

20. The system of claim 16, wherein the first device and the second device are gateway devices configured to communicatively couple the first network to a second network different from the first network.

Images