SNMP support for remotely distributed devices

Abstract

The system includes multiple network access devices, which are distributed remotely and connected to at least one internet-connected wide-area wireless network. Each of the network access devices persistently runs its own device management program. The device management programs are capable of reading and modifying local device status and maintaining their respective persistent network socket connections. The system includes an internet-connected data center with one or more servers running multiple Simple Network Management Protocol (SNMP) agent programs. Each SNMP agent program communicates with its respective device management program via a persistent network socket connection. The SNMP agent programs provide SNMP interfaces in the data center. A network management station manages its respective network access devices via its respective SNMP interface, its respective device management program, and its respective persistent network socket connection.

Description

Technical Field

[0001] Cross-Reference to Related Applications This application claims the benefit of Application No. 63 / 532,937 filed on August 16, 2023, the entire disclosure of which is incorporated herein by reference.

Summary of Invention

[0002] The present application generally relates to devices that facilitate network access at remote locations and devices (e.g., repeaters) that can extend wireless network access at a location. In one embodiment, the system includes a plurality of network access devices remotely distributed and coupled to at least one internet-connected wide area wireless network. Each of the plurality of network access devices persistently runs a respective device management program. The device management program is operable to read and modify local device status and maintain a respective persistent network socket connection. The system includes an internet-connected data center having one or more servers running a plurality of Simple Network Management Protocol (SNMP) agent programs. Each of the SNMP agent programs communicates with a respective one of the device management programs via the persistent network socket connection. The SNMP agent programs provide an SNMP interface at the data center. A network management station manages each respective network access device via the respective SNMP interface, the respective device management program, and the respective persistent network socket connection.

[0003] In another embodiment, the method can operate on a network access device operating remotely from a data center. The method includes connecting the network access device to an internet-connected wide-area wireless network. A persistent network socket connection is established over the wireless network to a Simple Network Management Protocol (SNMP) agent program via a persistently running device management program operating on the network access device. The SNMP agent program operates in the data center and manages SNMP services on behalf of the network access device. Based on commands received via the persistent network socket connection, the status of the network access device is read and / or modified via the device management program. In response to a command, status data is sent to the SNMP agent program, which uses the status data to communicate the status of the network access device to a network management station via SNMP.

[0004] In another embodiment, the method is operable via a data center. The method includes receiving connection requests from persistently running device management programs operating on each network access device. The network access devices are connected to the data center via one or more internet-connected wide-area wireless networks. Connection requests establish persistent network socket connections to the data center. The persistent network socket connections are connected to a multithreaded socket server that interfaces to a dedicated SNMP agent program on behalf of each network access device. SNMP commands are initiated via the SNMP agent program. The commands cause the respective device management programs to send corresponding non-SNMP commands via the persistent network socket connections. In response to the non-SNMP commands, non-SNMP status data is received by the SNMP agent program, which uses the non-SNMP status data to communicate the status of each network access device to the network management station via SNMP.

[0005] The figures and the following detailed description illustrate a typical embodiment in more detail. [Brief explanation of the drawing]

[0006] The following explanation will refer to the following diagram.

[0007] [Figure 1] Figure 1 is a block diagram of a system according to an exemplary embodiment. [Figure 2] Figure 2 is a block diagram of a network access device according to various exemplary embodiments. [Figure 3] Figure 3 is a block diagram of a network access device according to various exemplary embodiments. [Figure 4] Figure 4 is a block diagram of a network access device according to various exemplary embodiments. [Figure 5] Figure 5 is a block diagram of a network access device according to various exemplary embodiments. [Figure 6] Figure 6 is a block diagram showing details of a data center according to an exemplary embodiment. [Figure 7] Figure 7 is a block diagram illustrating message passing between a network access device and an agent program according to an exemplary embodiment. [Figure 8] Figure 8 is a flowchart of the method according to an exemplary embodiment. [Figure 9] Figure 9 is a flowchart of the method according to an exemplary embodiment.

[0008] The figures are not necessarily to scale. Similar numbers used in the figures refer to similar components. However, it should be understood that the use of numbers to refer to components in a given figure is not intended to limit the components in another figure labeled with the same number. [Modes for carrying out the invention]

[0009] The embodiments disclosed herein relate to architectures for supporting the control, monitoring, and processing of a scalable number of network access devices. Generally, a network access device in one embodiment is a dedicated device that can function as a local access point to a wide area network (WAN) and / or a global area network (GAN), the latter of which is the Internet. For example, a network access device may include a modem suitable for embedding in vehicles such as commercial vehicle fleets or in fixed locations such as weather stations. A wireless repeater, such as a 5G repeater, may also be considered a network access device. Some network devices may include other functions, such as routers, gateways, firewalls, and Dynamic Host Configuration Protocol (DHCP) servers, but this is not required. A network access device may provide high-speed network access over older networks such as 5G cellular networks or Long-Term Evolution (LTE) 4G networks. In some embodiments, a network access device may access the network through other network means, such as cellular tethering. In other words, a network access device may provide Internet access by itself or through another network access device.

[0010] Depending on the target use case, the hardware used for network access devices may include a low-power (e.g., around 80 MHz) embedded 32-bit processor, or a computing system with higher processing power and memory that supports an operating system and application stack (e.g., OpenWRT with a MIPS or ARM processor). The former may be desirable for maintaining relatively slow network connections with a limited power budget (e.g., battery-powered), while the latter may be desirable for embedded systems that support faster speeds and multiple users, at the expense of (particularly) relatively higher power consumption. In these examples, the available services and features may vary between device architectures, and embedded devices may include minimal features, in particular, to achieve low cost and low power consumption.

[0011] Organizations may employ numerous network access devices that are geographically dispersed and have diverse configurations. This can complicate the management of these devices, for example, by determining their status / alerts and applying changes. Protocols, particularly the Simple Network Management Protocol (SNMP), have been developed to manage devices in this manner. Managed devices (e.g., computers, routers, printers) often include an SNMP agent, which collects data from the managed devices and communicates with a network management station (NMS) on the network. Using SNMP, network administrators can manage and monitor numerous SNMP devices from a single interface.

[0012] Organizations such as businesses, schools, and governments often employ SNMP to manage their network devices. However, these organizations typically have complete control over their local networks. For organizations with geographically separated local networks, the networks can be connected together via virtual private network (VPN) tunnels or similar. VPN tunnels allow different local networks to connect securely and reliably to each other over insecure channels such as the public internet. Even so, implementing SNMP monitoring and control systems across multiple distributed networks is no trivial matter, as tunnels must be properly configured to ensure that SNMP traffic is reliably routed in both directions through multiple subnetworks.

[0013] For smaller network access devices, a VPN tunnel providing a full Internet Protocol (IP) stack may not be practical because the devices may have limited processing power, and the VPN protocol uses significant overhead that may not be readily available in wireless large area networks. Nevertheless, it is desirable to implement SNMP (or other similar network management protocols) for a collection of remotely distributed network access devices. This can avoid costly on-site visits to resolve issues, reconfigure, and perform similar tasks. Furthermore, although the SNMP protocol is not necessarily optimized for small mobile devices, its widespread adoption makes it a desirable choice for managing this type of device.

[0014] Embodiments described herein provide SNMP support for monitoring and controlling network access devices. This support includes trapping, which involves asynchronous alerts sent from an SNMP agent to an NMS. Network devices utilize long-range wireless data access protocols over cellular standard networks, such as 3G, 4G, 5G, LTE, and Edge. These wireless networks allow packet-switched IP traffic to pass through, but may not permit certain traffic to pass through in certain directions. For example, Transmission Control Protocol (TCP) SYN IP packets may not be permitted to pass from the Internet to a mobile station in a wireless network, but SYN packets may be permitted to pass through in other directions, such as connecting a mobile station to an Internet server.

[0015] To effectively manage a large number of network access devices, network service elements (e.g., processes running on a server with an internet connection) should be able to send specific commands to network access devices in real time or near real time (e.g., with a round-trip time of approximately 3000ms or less) and receive responses to those commands within a comparable time frame. This would allow, for example, an engineer at one end of a country to monitor network devices (for optimal performance) while a technician in another region of the country moves and / or aims the network devices towards a base station.

[0016] Furthermore, as described below in this specification, real-time access means being kept alive to receive commands asynchronously (e.g., best effort to maintain operational status and network connectivity at any time of day). If the device is permitted to disconnect from the server in some cases, the response time will increase by at least the time required to establish a new connection. Embodiments of network access devices described herein maintain a persistent internet socket connection, thereby reducing these types of delays in device-server communication.

[0017] Furthermore, real-time access to network access devices enables the processing of alarms (generating traps in SNMP) when they occur. Real-time control allows network access devices to be put into sleep mode on demand. Situations occurring in devices that cause interference or abnormal behavior can be handled in a timely manner via real-time network communication. In addition, devices can be configured in real time for optimizations such as beam steering and positioning while monitoring RSSI. Polling of device sensor data such as temperature without rescheduling is another advantage of real-time access. The following disclosure describes embodiments that can be used to effectively manage a large number of network access devices.

[0018] In Figure 1, a block diagram shows a system 100 according to an exemplary embodiment. The system 100 includes a plurality of network access devices 102, which are geographically distributed and connected to at least one internet-connected wide-area wireless network. The network access devices 102 are shown in Figure 1 with a LAN interface on the left, although LAN interfaces (e.g., Ethernet, WiFi) are not used in all embodiments. In this example, two separate wireless provider networks 104 and 106 are shown, each connected to the internet 108. Each of the plurality of network access devices 102 persistently runs its own device management program 110. The device management program 110 is operable to read and modify, in particular, the local device status, which is the status of the device 102 on which the device management program 110 operates. The device management program 110 maintains a persistent network socket connection 112 with an internet-connected data center 114.

[0019] The data center 114 includes one or more servers 116 running multiple Simple Network Management Protocol (SNMP) agent programs 118. Each SNMP agent program 118 communicates with its respective device management program 110 via a persistent network socket connection 112. The SNMP agent programs 118 provide an SNMP interface 118a in the data center 114. The network management station 120 manages its respective network access devices 102 via its respective SNMP interface 118a, device management program 110, and its respective persistent network socket connection 112, for example, a TCP / IP socket.

[0020] Generally, a persistent network socket connection 112 is a logical network representation that comprehensively represents the underlying network and physical links, such as wired or wireless network media and protocols. A network socket connection 112 is persistent in the sense that the device management program 110 makes best efforts to keep the socket connection 112 (or equivalent socket connection) continuously open. For example, a persistent network socket connection 112 may be unintentionally disconnected due to signal loss. In such an event, the device management program 110 attempts repeated attempts to re-establish the network socket connection 112. Since this re-establishment may change some logical values ​​of the socket connection (e.g., port number, source and destination IP addresses), the re-established socket may, in a sense, be considered a different socket. A TCP / IP set is generally defined by the number of two sets, each set in the form of {IP address, TCP port}. Each set defines a communication channel, one from client to server and the other from server to client.

[0021] In some implementations, the network access device 102 does not enter a sleep mode or a standby mode. Furthermore, the device 102 is configured to always keep the device management program 110 in an operating state or keep it in a permanently executed state as described above. Accordingly, the persistent network socket connection 112 should be available whenever possible. Some socket parameters may need to change over time (e.g., the network access device 102 may move to a different carrier network and re-establish a socket connection with a different remote IP address), therefore, a universally unique identifier such as a media access control (MAC) address or a custom identifier may be used to define the persistent network socket connection 112 at a high level. This allows specific TCP / IP socket parameters to be changed over time while the unique identifier remains unchanged, which facilitates an efficient method for tracking and retrieving data, for example, by using the identifier as a database primary key in a table storing data of the network access device 102.

[0022] The SNMP agent program 118 can utilize a server environment including high availability, abundant computing resources, ease of management, and the like. As such, the experience of an organization monitoring a system such as that represented by the terminal 122 is different from that of conventional SNMP implementations that monitor local enterprise devices. The SNMP agent program 118 can be programmed to account for the types of problems associated with remote devices operating over untrusted data links, which is not anticipated for typical SNMP implementations. For example, some data can be cached by the SNMP agent program 118 and used to respond to some queries even when the associated network access device 102 is not currently reachable.

[0023] Another advantage provided by the SNMP agent program 118 relates to the general-purpose device interface 118b used to communicate with the network access device 102 via the persistent network socket connection 112. This general-purpose interface 118b facilitates communication via a common protocol over the connection 112, as a result of which a single code base can be used for the SNMP agent program 118 to manage various network access devices 102. This differs from common implementations of SNMP agents, in which device status tracking code is tightly coupled to the hardware monitored by the agent.

[0024] In FIGS. 2 to 5, block diagrams show network access devices that may be used in the system 100. In FIG. 2, the block diagram shows a network access device 200 according to an exemplary embodiment. The network access device 200 is suitable for providing fixed network support as indicated by the local area network (LAN) interface 202. The network access device 200 connects to an Internet-connected wide area wireless network as indicated by the antenna 204. The wireless network may be a high-speed network such as a 4G / 5G mobile data network, but the network access device 200 may also be adapted to other networks, for example, 2G Edge networks and satellite data networks.

[0025] The network access device 200 may often include what is considered a complete computing hardware and software stack, as indicated by component 206. The hardware is indicated by block 208, which includes a processor, e.g., a central processing unit (CPU), memory, and input / output (I / O). In some cases, the network access device 200 includes an operating system (OS) with a kernel 209 that provides low-level hardware access. In other cases, an embedded program (e.g., in C) provides low-level hardware access. A services and hardware abstraction layer (HAL) 210 interfaces with the kernel 209, which enables higher-level programs to access hardware and other services through a common interface, e.g., an application program interface (API). While the kernel 209 and services / HAL 210 are often considered together as an OS, a particular OS may have more or fewer elements, or the OS may not be used at all, with an embedded program providing the interface instead.

[0026] Shell 211 represents a user program that runs within the OS, interacts with the user, and / or may run in background or batch mode. Generally, shell 211 is a general-purpose program that provides a user environment for launching a specific end-user program. Examples of shells include Unix® shells (sh, bash, csh). User programs include device management programs 212, which are described in more detail below.

[0027] The network access device 200 can provide LAN access to the Internet at the customer's location. Therefore, other user programs, in addition to the device management program 212, may be part of the components 206 incorporated into the device 200, enabling the device 200 to perform functions such as a router, gateway, firewall, DHCP server, DNS cache, etc. In some embodiments, the OS may include a Linux® kernel with GNU core utilities. The GNU / Linux OS is robust, secure, highly configurable, and runs on a wide variety of hardware. The GNU / Linux OS has a native IP network stack including TCP / IP and UDP / IP. The network access device 200 may also include other hardware not shown, such as a 5G / LTE modem, Ethernet controller, power management controller, etc.

[0028] The device management program 212 supports real-time, two-way TCP / IP connections to a dedicated SNMP agent program 118 in an internet-accessible data center 114. The device management program 212 may be written in a language such as C to generate efficient native machine code, but other programming languages ​​and non-native code running on a virtual machine may also be used. The device management program 212 is activated at startup, for example, as a background daemon. Generally, a daemon is a background program intended to run continuously or be started automatically as needed. Similar functionality to a daemon is provided in the Windows® operating system by something called a "service."

[0029] In a Linux daemon implementation, a special program (e.g., a watchdog shell script) can keep the device management program 212 running indefinitely, even if the TCP / IP connection is lost or the device management program crashes. The device management program 212 establishes a TCP / IP connection to the data center 114, to which the SNMP agent program 118 connects. The device management program 212 also interfaces with hardware and software services on the network access device 200 to provide status data that can be used by the SNMP agent program 118. For example, the device management program 212 may use a GNU / Linux utility such as "lm-sensors" to determine the CPU temperature. Similar utilities may be used to access other hardware, such as a data modem, to determine relevant information about the data link, such as signal strength, the channel / band being used, and the amplifier status. While parts of the device management program 212 that communicate with the data center 114 may be generic, parts that interface with the network access device 200 are customized for specific types and versions of hardware and software on the device 200.

[0030] In Figure 3, a block diagram shows a network access device 300 according to another exemplary embodiment. The network access device 300 supports mobile and fixed stations, for example, via a high-speed connection through an LTE modem 302. The network access device 300 may also be referred to as an Ethernet injector that acts as a bridge between the LTE modem 302 and the Ethernet port, and the port on a computer 304. The network access device 300 may be connected to the modem 302 via a Universal Serial Bus (USB). The computer 304 can access the network provider's 4G LTE wireless network by connecting to the Ethernet port. In other embodiments, a WiFi router, an Ethernet switch, etc., may be connected to support multiple network devices.

[0031] The network access device 300 supports an embedded Linux OS with full TCP / IP stack support, as indicated by component 306. Like device 200 in Figure 2, device 300 in Figure 3 includes a device management program 308, which monitors the local device status and communicates this status to an SNMP agent program 118 operating in an internet-accessible data center 114. The device management program 308 has similar features to those described in relation to program 212 in Figure 2, such as maintenance of persistent socket connections, background operation as a daemon, and a watchdog application or script for starting and maintaining the daemon operational. Because the hardware and / or software of the two devices 200 and 300 may differ, the device management programs 212 and 308 are configured differently to take this into account. Nevertheless, both device management programs 212 and 308 have the same general-purpose interface for connecting to a dedicated SNMP agent program 118. Furthermore, both device management programs 212 and 308 can rely on operating systems such as Linux to provide a significant amount of functionality.

[0032] In Figure 4, a block diagram shows a network access device 400 according to another exemplary embodiment. The network access device 400 is configured to access the internet, for example, via a peripherally mounted LTE modem 402. The LTE modem 402 is configured via AT commands. The LTE modem 402 supports TCP / IP connections to TCP / IP servers over the internet. The network access device 400 is an embedded system, as indicated by component 406, which includes a system-on-a-chip (SoC) 408, a basic firmware interface indicated by a kernel / API 410, and an embedded device management program 412 operating in the system firmware. Since the kernel / API 410 does not typically have features and services associated with an OS, the management program 412 is referred to herein as embedded object code that operates in an embedded system without an operating system. The network access device 400 interfaces with a radio repeater 404 (e.g., a 5G repeater) via, for example, an RS-232 serial interface (or SPI interface). SoC408 gains access to all parameters of repeater 404 and can set any parameters of the repeater, including beam steering, network provider base station RSSI, etc. In this case, 400 can provide complete two-way real-time control of the repeater via SNMP agents through NMS.

[0033] The block diagram in Figure 5 provides more information about the network access device 400. The LTE modem 402 interfaces with the SoC 408 via a Universal Asynchronous Receiver / Transmitter (UART) interface 500, also known as a Serial Peripheral Interface. The device management program 412 for this device 400 is called the Remote Command Unit (RCU). The RCU program runs on the SoC 408 and interfaces with the peripherally mounted LTE modem 402 and wireless repeater 404 via their respective UART interfaces 500 and 502. An API exists for the RCU firmware to control the wireless repeater 404 and obtain information via the UART interface 502.

[0034] Furthermore, the network access device 400 has an electrically erasable programmable read-on rememory (EEPROM) 504, which stores the IP and TCP port of the TCP / IP server in the data center where the network access device 400 establishes a TCP / IP connection via the LTE modem 402. The RCU subsystem has a unique identifier (e.g., a 48-bit binary value) that identifies the network access device 400 to enterprise entities for identification purposes via the SNMP agent program 118. The unique identifier is also stored in the EEPROM 504. Since some network interface devices may not have an equivalent unique address, the unique identifier is used in place of a fixed hardware address, such as a MAC address. Ethernet devices have MAC addresses, but some network access devices (such as this device 400) do not have an Ethernet interface. Even if a network interface device has a unique hardware address, some network interfaces (e.g., modems, network interface cards) do not. Therefore, hardware changes require the monitoring and control system to be reconfigured to take the new hardware address into account in such cases. By being stored in EEPROM 504, the unique identifier remains with the network access SoC 408 even when peripheral hardware is changed, and the unique identifier can be retrieved via EEPROM 504 even if the modem fails, which may not be the case if the ID stored in the modem is used. Additionally, the Wi-Fi access point (AP) 506 and serial port 508, which can be used for local control and configuration of the network access device 400 and repeater 404, for example, initial setup and field maintenance, are recognized in Figure 5.

[0035] In Figure 6, a block diagram shows an embodiment of a data center 114 according to an exemplary embodiment. The data center 114 may include enterprise-level architecture features such as distributed computing, large-capacity storage, redundancy, and high availability. The data center supports network access devices 102, including any device type as described above in Figures 2 to 5. The network access device 102 initiates and establishes TCP / IP socket connections with internet-facing servers, referred to herein as multithreaded socket servers 600. The multithreaded socket servers 600 may include multiple servers, such as load balancing servers, which accept connections from the device management program 110 of the network access device 102 via the internet 108, hand off the connections to another software module or machine, which then manages the connections.

[0036] The multithreaded socket server 600 creates socket threads 602 that connect each device management program 110 to its dedicated SNMP agent program 118. The SNMP agent program 118 initiates and establishes a TCP / IP connection to the multithreaded socket server 600. The socket threads 602 are used to provide links from the network access devices 102 to their respective SNMP agent programs 118. As the name suggests, the socket threads 602 may be launched as lightweight processes from a single server program, thereby minimizing the use of computer resources by a large number of threads 602. However, in some embodiments, the socket threads 602 may be standalone processes, each having a separate process ID in the operating system, for example.

[0037] As described above, each SNMP agent program 118 is associated with its respective network access device 102, for example, in a one-to-one mapping. The SNMP agent program 118 has an SNMP interface 118a and a device interface 118b. The device interface 118b uses a general-purpose message passing interface, which is described below and can be implemented in all versions and types of the device management program 110. Both the SNMP agent program 118 and the NMS 120 can operate on the same server or subnet to keep SNMP traffic locally away from the internet 108.

[0038] Due to the nature of the network connection, the address and port that define the TCP / IP socket connecting the SNMP agent program 118 to the network access device 102 may change over time. For example, the destination IP and port number on the network access device 102 (connected to the device interface 118b of the agent program 118) may be changed through the SNMP agent program 118, allowing the network access device 102 to connect to different servers or data centers to facilitate expansion and / or load management. The SNMP interface 118a may use the same local IP and port in the data center 114 regardless of whether the socket parameters of the network access device 102 on the SNMP interface 118a change.

[0039] Because the IP address and port of the socket used to communicate between the SNMP agent program 118 and the network access device 102 can change, using socket parameters (e.g., IP address) to uniquely identify each device 102 in the system can be complex. Therefore, the SNMP agent program 118 and NMS 120 are associated with a unique identifier, such as a MAC address or 48-bit identifier not associated with a network interface, as described above. The unique identifier categorizes data transfers along the entire link from the network access device 102 to the SNMP agent program 118. Therefore, the unique identifier can be used, for example, as an indicator of data about the network access device 102 in the database 606.

[0040] Database 606 may be configured as a relational database, such as MySQL®, Actian® X, or Microsoft® SQL Server. Such a database may be associated with a set of Structured Query Language (SQL) commands and may be accessed simultaneously by multiple clients locally or over a network. In the system shown, database 606 may include a table for network access devices 102, the primary key of which may be a unique identifier. The device table may include, in particular, a unique identifier for device 102 and its associated SNMP agent program 118, the address and port of the device-server socket, the GPS coordinates of network access device 102, a description of network access device 102 (e.g., a specific building in a specific city), the wireless network provider used by device 102, and the online status of network access device 102. Other tables in database 604 may be used for SNMP-specific purposes, such as having a table to log all alarms, such as when device 102 goes offline or online, or heartbeat timeout alarms generated by server 600 and / or SNMP agent program 118.

[0041] It should be noted that some network access device data that does not change frequently may be cached by the SNMP agent program 118 and stored in the database 604. Examples of such data include labels / descriptors, last known geographical locations, non-dynamic repeater parameters, and parameters that do not change unless configured. In some embodiments, the SNMP agent program 118 caches first status data associated with each network access device 102. The cached first status data is used to respond to the NMS 120 when the persistent network socket connection 112 is down. Second status data (different from the first status data) for each network access device 102 is not cached. The second status data may include data that changes periodically, such as signal strength measurements. In response to the NMS 120 requesting the second status data, the SNMP agent program 118 may provide an indication that the persistent network socket connection 112 is down, or that some other error has occurred (e.g., the connection can send pins, but the device management program 110 is not responding). The unique ID is not cached and, in addition to the heartbeat and PIN, is used to indicate whether the device is connected (when a request for the unique ID is not valid).

[0042] The SNMP agent program 118 may be implemented to run on the Java® Virtual Machine and may utilize features of Java Enterprise Edition (EE). Java EE provides a platform with enterprise features such as distributed computing and web services. For example, Java EE intrinsics can automatically instantiate lightweight objects (e.g., Java Beans) that map to database objects and can be used to perform tasks such as processing network requests. Java EE also manages database connections via Java Database Connectivity (JBDC), which efficiently manages a large number of database connections and provides a way for applications to access, add to, and modify database data. The SNMP agent program 118 may be implemented in the Java EE framework in the server 116 and other components shown in the data center 114. Note that other frameworks, such as Microsoft .NET or C++, may be used instead of Java to implement the data center components shown.

[0043] Another capability provided by Java is the ability to build dynamic web content via technologies such as Java Servlets, Java Server Pages (also known as Jakarta Server Pages), etc. This is demonstrated by the web interface 608, which allows terminal devices 610 to monitor the status of software components in the data center 114, including the SNMP agent program 118, socket threads 602, and multithreaded socket server 600. The web interface is implemented in Java and may run on a server such as Oracle® GlassFish or Apache Tomcat®. The capabilities provided by the web interface 608 are different from those provided by the NMS 120, although some data provided by both, such as remote device status, may overlap. In general, the web interface 608 allows system administrators to monitor components of the data center 114. In contrast, the NMS 120 allows existing SNMP monitoring tools to monitor network access devices 102 without requiring any knowledge of the intervening services provided by the data center 114.

[0044] Furthermore, other software may be used to monitor data center components in addition to or separate from the web interface. An example of this other software is a Java application 612 for monitoring and controlling network access device 102. Java application 612 interfaces with network access device 10 through SNMP agent program 118. In this way, developers can fully test and verify the SNMP agent performance to prepare the SNMP solution for deployment. Note that if another framework, such as .NET, is used to implement the data center components, application 612 may use this other framework instead of Java.

[0045] The Java application may provide a graphical user interface (GUI) 614 that facilitates charting parameters such as the Received Signal Strength Indicator (RSSI) and temperature in strip charts. Data from the charts can be retrieved at regular intervals via SNMP "get" commands. The Java application 612 has few limitations on its user interface and capabilities, although this is not always the case with respect to the web interface 608. The Java application 612 also accesses the database 606, for example, via JDBC. The Java application 612 provides a list of network access devices 102 and then connects to an SNMP agent program 118 to monitor and control the devices 102. The Java application 612 can also chart alarms and send messages (e.g., email, SMS text messages) to a list of affected personnel.

[0046] Any enterprise entity within the data center 114 may establish a direct connection to the network access device 102 through an application programming interface (API) specific to the server 600, along with a unique identifier. Once a connection is established to the device management program 110 running on device 102, the enterprise entity may issue commands and receive responses containing the requested information. Commands may also be sent to activate / deactivate subsystems of the network access device 102 by changing parameters, such as commands to take the device offline, commands to put the device into a stopped state (e.g., low-power mode), and commands to restart the device.

[0047] The enterprise architecture provides the SNMP agent program 118 with the ability to establish a direct link to the network access device 102 and support get / set commands containing real-time data. The SNMP agent can also access tables in database 606 for the entire category of SNMP get / set commands. A server API enables real-time processing of alarms (traps) when they occur. The SNMP agent program 118 and / or socket server 600 can monitor the connection status of the network access device 102 using polling messages over a TCP / IP connection. If the network access device 102 does not respond to multiple polling messages, the SNMP agent program 118 issues a trap with the device's unique ID and offline status. Also, when device 102 is online and connected, alarms initiated by the device management program 110 are sent to the SNMP agent program 118 via a generic device-agent interface 118b.

[0048] Each SNMP agent connects to a unique UDP port for communication between the SNMP agent and the NMS. The unique port, which is given a unique ID for the remote device, is obtained from the remote device table in database 606. By using the SNMP agent program 118, the network access device 102 may have multiple connections from multiple enterprise entities (e.g., NMS 120, web interface 608, application 612) via the data center 114. For scalability, each network access device 102 has the IP address and TCP port of the multithreaded socket server 600 to which each network access device 102 connects. When a large number of network access devices 102 are deployed, the large number of network access devices 102 can be configured to face different servers to enable isolation and, therefore, scalability.

[0049] Each server 600 may support thousands of SNMP agent programs 118, one for each remote network access device 102. Each server rack may supply a collection of network access devices 102. The IP / port used by the network access device 102 to connect to the data center 114 can be changed using the SNMP "set" command, and then instructed to use the new IP / port to connect to at startup. This can also be done on-site or in the factory before deployment using the local interface (e.g., WiFi, serial port) on the network access device 102. Other device settings can also be set or changed using SNMP "set" commands sent from the NMS to the SNMP agent program 118. The SNMP "set" command can set or change settings such as gain per radio frequency (RF) path, IP address and TCP port for the server, unique identifier, status light, authentication data for the wireless network and data center, and network configuration data such as DNS servers. The SNMP "set" command can also trigger events such as entering suspend mode and restarting the device. In another example, the SNMP "set" command can steer a beam in a repeater.

[0050] Furthermore, the SNMP agent program 118 supports reading device status via "get" commands sent from the NMS 120 to the agent 118. For example, SNMP "get" commands may be used to read RF front-end gain settings, RSSI for each RF path, unique ID, power amplifier temperature in the RF front-end, SoC operating environment status (e.g., temperature, voltage), GPS coordinates, beamforming steering support, etc. The SNMP agent program 118 can also configure SNMP traps such as exceeding a temperature threshold, losing connectivity to device 102 over the Internet, and device 102 being online over the Internet. This is a non-exclusive example of SNMP get / set commands and traps.

[0051] A text-based Management Information Base (MIB) file and one or more object identifiers (OIDs) are associated with SNMP commands and traps. Commercial-grade MIB compilers may be used to develop MIBs, compile them, and integrate them into SNMP agent programs 118. An example OID that may be used for a network access device is 1.3.6.1.4.1.58864.1.1.1400.2.1, where 58864 is a vendor-private enterprise number (PEN) as provided by the Internet Number Assignment Authority, and 1400 is an internal device designator assigned by the vendor. This is similar to how MAC addresses are assigned to ensure that MAC addresses are globally unique (if properly implemented by the vendor).

[0052] As described above, the SNMP agent program 118 and the device management program 110 of the network access device 102 communicate using a general-purpose message passing protocol over the persistent network socket 112. In Figure 7, the block diagram shows command exchange according to an exemplary embodiment. The SNMP agent program 118 exchanges SNMP data 700 with entities in the data center 114. The SNMP data 700 includes SNMP commands, responses, and traps. The device management program 110 exchanges device data 702 within the device 102, for example, via the OS, firmware, direct memory access, etc.

[0053] The command 704 sent from the SNMP agent program 118 and the corresponding response 706 sent from the network access device 102 are shown in Figure 7. Each command has an associated response. Response 706 is a name-value pair ending with a semicolon (;), comprising a command part 706a, an equals sign (=), and response 706b. In this way, a command may arrive through multiple attempts if necessary, and a response may be checked to verify (using the name-value pair) that the response is from a command. For example, a command such as "Get_Temp" may call the response "Get_Temp=30" rather than just "30". This can prevent, for example, improper interpretation of responses when multiple commands are sent rapidly in succession. Commands and responses may also include and / or be associated with an identifier that is unique to each network access device 102. If a response is not received within a predetermined time, the SNMP agent program sends an SNMP trap indicating that the network access device is offline.

[0054] The SNMP agent program 118 (or multithreaded socket server 600) sends a heartbeat (as previously described as a polling message) to the network access device 102 to request a response. The network access device 102 responds to the heartbeat, indicating that device 102 is alive and has a good connection. If the response is not received at the data center 114 within a certain interval (e.g., 1 minute), an alarm is sent. The database table of the network access device's online status is also updated to show as offline.

[0055] In Figure 8, a flowchart illustrates a method according to an exemplary embodiment applicable to a network access device operating remotely from a data center. The method includes connecting the network access device to an internet-connected wide-area wireless network (800). A persistent network socket connection is established on the wireless network to an SNMP agent program via a persistently running device management program operating on the network access device, which operates in the data center and manages SNMP services on behalf of the network access device (801). Reading and / or modifying the status of the network access device is performed via the device management program based on commands received via the persistent network socket connection (802). In response to a command, status data is sent to the SNMP agent program (803), which uses the status data to communicate the status of the network access device to a network management station via the SNMP protocol.

[0056] In Figure 9, a flowchart illustrates a method according to an exemplary embodiment, which is a method that can operate through a data center. The method includes receiving connection requests from persistently running device management programs operating on each network access device (900). The network access devices are connected to the data center via one or more internet-connected wide-area wireless networks. Connection requests establish persistent network socket connections to the data center. The persistent network socket connections are connected to a multithreaded socket server that interfaces to a dedicated SNMP agent program on behalf of each network access device (901). SNMP commands are initiated via the SNMP agent program, which causes the respective device management programs to send corresponding non-SNMP commands via the persistent network socket connection (902). In response to the non-SNMP commands, non-SNMP status data is received by the SNMP agent program, which uses the non-SNMP status data to communicate the status of each network access device to the network management station via SNMP (903).

[0057] In summary, a system, method, and apparatus for monitoring and controlling multiple network access devices are described, which are remotely distributed and coupled to at least one internet-connected wide-area wireless network. Each of the network access devices persistently runs its own device management program. An internet-connected data center operates multiple SNMP agent programs. Each SNMP agent program communicates with the respective device management program via a persistent network socket connection. Providers employing the various embodiments described herein may have real-time, two-way control of the network access devices for design and network optimization. Other features provided by the embodiments described herein include adaptability to a variety of remote network access devices and device capabilities. Furthermore, since the socket connection is from the remote device to the data center and not the other way around, the system is not affected by internet service provider (ISP) blocking of connections to local devices.

[0058] References to the accompanying set of drawings forming part of this disclosure are made herein, but at least those skilled in the art will understand that various adaptations and modifications of the embodiments described herein are within the scope of this disclosure or do not depart from the scope of this disclosure. For example, the aspects of the embodiments described herein may be combined with one another in various ways. Accordingly, it should be understood that within the claims of the appended claims, the claimed invention may be carried out in ways other than those expressly described herein.

[0059] All references and documents enumerated herein are expressly incorporated herein by reference in their entirety, except to the extent that such references and documents would directly contradict this disclosure. Unless otherwise indicated, all numbers used in the specification and claims to represent the size, quantity, and physical properties of features may be understood to be modified with the terms “exactly” or “approximately.” Thus, unless otherwise indicated, the numerical parameters described herein in the specification and appended claims are approximate, which may vary depending on the desired properties to be obtained by a person skilled in the art utilizing the art disclosed herein, or, for example, within a typical range of experimental error.

[0060] Numerical ranges indicated by endpoints include all numbers contained within that range (for example, 1 to 5 includes 1, 1.5, 2, 2.75, 3, 3.80, 4, and 5) as well as any range within that range. In this specification, the terms "up to" or "less than" a number (for example, up to 50) include the number (for example, 50), and the terms "greater than" a number (for example, 5 or more) include the number (for example, 5).

[0061] The terms “bonding” or “connecting” refer to elements being attached to each other directly (in direct contact with each other) or indirectly (with one or more elements in between, or by attaching two elements). To describe that the bonding or connection is configured to allow the components to interact in order to perform at least some function, either term may be modified with “operationally” and “operationally capable,” which may be used interchangeably (for example, a wireless chip may be operationally bonded to an antenna element to provide radio frequency electrical signals for wireless communication).

[0062] References to “one embodiment,” “embodiment,” “a particular embodiment,” or “some embodiments” mean that a particular feature, configuration, composition, or characteristic described in relation to an embodiment is included in at least one embodiment of this disclosure. Therefore, the appearance of such phrases in various places throughout this disclosure does not necessarily refer to the same embodiment. Furthermore, a particular feature, configuration, composition, or characteristic may be combined in any preferred manner in one or more embodiments.

[0063] The terms “preferred” and “preferred” refer to embodiments of the disclosure that may offer certain advantages under specific circumstances. However, other embodiments may also be preferred under the same or other circumstances. Furthermore, the description of one or more preferred embodiments does not imply that other embodiments are unhelpful, nor is it intended to exclude other embodiments from the scope of the disclosure.

[0064] When used herein and in the appended claims, the singular forms “a,” “an,” and “the” include embodiments having multiple references unless otherwise specifically defined in the content. When used herein and in the appended claims, the term “or” is generally adopted to mean “and / or” unless otherwise specifically defined in the content.

[0065] As used herein, “have,” “possess,” “include,” “contain,” “equip,” “equip,” and similar terms are used in their open-ended sense and generally mean “include, but not limited to.” “Substantially consist of,” “composed of,” and similar terms will be understood to be encompassed by “equip” and similar terms. The terms “and / or” mean one or all of the enumerated elements, or a combination of at least two of the enumerated elements.

[0066] The phrases “at least one of,” “encompassing at least one of,” and “one or more of” that follow a list refer to any one item in the list, or any combination of two or more items in the list.

Claims

1. A system, wherein the system is A plurality of network access devices that are remotely distributed and connected to at least one internet-connected wide-area wireless network, each of the plurality of network access devices persistently runs its own device management program, and the device management program is Read and change the local device status. Multiple network access devices capable of maintaining their respective persistent network socket connections, An internet-connected data center comprising one or more servers running multiple Simple Network Management Protocol (SNMP) agent programs, each of the SNMP agent programs communicating with each of the device management programs via the persistent network socket connection, and the SNMP agent programs providing an SNMP interface in the data center, A network management station manages each network access device via its respective SNMP interface, its respective device management program, and its respective persistent network socket connection. A system that includes these features.

2. The system according to claim 1, wherein at least one of the device management programs includes a daemon that runs native object code in the operating system.

3. The system according to claim 2, wherein at least one of the device management programs comprises embedded object code that operates in an embedded system without a local operating system, and the native object code and the embedded object code communicate via a common protocol in the persistent network socket connection between the native object code and the embedded object code, respectively.

4. The system according to claim 1, wherein at least one of the device management programs comprises embedded object code that operates in an embedded system without an operating system.

5. The system according to claim 4, wherein the embedded object code establishes and maintains the persistent network socket connection via a wireless modem mounted around the network access device.

6. The system according to claim 5, wherein the wireless modem and network access device mounted on the periphery are coupled via a serial peripheral interface.

7. The system according to claim 1, wherein the internet-connected data center further comprises a multithreaded socket server, the multithreaded socket server establishes the persistent network socket connection and facilitates the connection of the persistent network socket connection to the SNMP agent program.

8. The system according to claim 1, wherein each SNMP agent program caches first status data associated with each of the network access devices, the cached first status data is used to respond to the network management station when the persistent network socket connection is down, and second status data for each of the network access devices is not cached when the persistent network socket connection is down, and the second status data is different from the first status data.

9. The system according to claim 8, wherein, in response to the network management station requesting the second status data, each SNMP agent program provides an indication that the persistent network socket connection is down.

10. The system according to claim 1, wherein the SNMP agent program communicates with each of the device management programs via a non-SNMP message passing protocol.

11. The system according to claim 10, wherein the non-SNMP message passing protocol comprises a command and an associated response, and the associated response comprises the command and a response to the command.

12. A method applicable to a network access device operating remotely from a data center, wherein the method is Connecting the aforementioned network access device to an internet-connected wide-area wireless network, The process involves establishing a persistent network socket connection in the wireless network to a Simple Network Management Protocol (SNMP) agent program via a persistently running device management program operating on the network access device, wherein the SNMP agent program operates in the data center and manages SNMP services on behalf of the network access device. Based on commands received via the persistent network socket connection, to read and / or modify the status of the network access device via the device management program, In response to the command, status data is sent to the SNMP agent program, and the SNMP agent program uses the status data to communicate the status of the network access device to the network management station via SNMP. Methods that include...

13. The method according to claim 12, wherein the network management station initiates an SNMP command using the SNMP agent program, and the SNMP agent program causes the command to be transmitted via the persistent network socket connection.

14. The method according to claim 12, wherein the SNMP agent program communicates with the device management program via a non-SNMP message passing protocol.

15. The method according to claim 14, wherein the non-SNMP message passing protocol comprises the command and associated response, and the associated response comprises the command and a response to the command.

16. A method that can operate via a data center, the method is Receiving connection requests from persistently running device management programs operating on each network access device, wherein the network access devices are connected to the data center via one or more internet-connected wide-area wireless networks, and the connection requests establish their respective persistent network socket connections to the data center. The persistent network socket connection is coupled to a multithreaded socket server that interfaces with a dedicated SNMP agent program on behalf of each of the aforementioned network access devices, This involves initiating an SNMP command via the SNMP agent program, wherein the SNMP agent program causes the corresponding non-SNMP command to be sent to the respective device management program via the persistent network socket connection. In response to the aforementioned non-SNMP command, the SNMP agent program receives non-SNMP status data, and the SNMP agent program uses the aforementioned non-SNMP status data to communicate the status of each of the network access devices to the network management station via SNMP. Methods that include...

17. The method according to claim 16, wherein the connection request is received by a multithreaded socket server, and the multithreaded socket server connects the persistent network socket connection to the multithreaded socket server which is interfaced to the SNMP agent.

18. The method according to claim 17, wherein the internet-connected data center further comprises a multithreaded socket server, the multithreaded socket server establishes the persistent network socket connection and facilitates the coupling of the persistent network socket connection to the multithreaded socket server interfaced to the SNMP agent.

19. The method according to claim 16, wherein the SNMP agent program caches first status data associated with each of the network access devices, the cached first status data is used to respond to the network management station when the persistent network socket connection is down, and second status data for each of the network access devices is not cached, and the second status data is different from the first status data.

20. The method according to claim 19, wherein, in response to a request for the second status data from the network management station, the SNMP agent program provides an indication that the persistent network socket connection is down.

21. The method according to claim 16, wherein the SNMP agent program communicates with each of the device management programs via a non-SNMP message passing protocol.

22. The method according to claim 21, wherein the non-SNMP message passing protocol comprises a command and an associated response, and the associated response comprises the command and a response to the command.

Images