High-performance LoRaWAN gateway based on Linux system and data processing method

Abstract

The invention relates to a high-performance LoRaWAN gateway based on a Linux system and a data processing method, and the gateway comprises the following service modules: a LoRa packet forwarding service which is used for initializing a LoRa radio frequency module based on a Semtech libragw library, setting radio frequency parameters such as a center frequency, a bandwidth and a spreading factor, and carrying out bidirectional data interaction with a Chirpstack service through a local UDP protocol. The gateway can still independently complete LoRaWAN data analysis and node management in a network-free environment, through local UDP optimization and caching, delay and packet loss caused by network fluctuation are reduced, dependence on cloud computing resources is reduced, a system architecture is simplified, deployment and operation and maintenance costs are reduced, data security and real-time performance are enhanced, and the method and the device are suitable for large-scale popularization and application. And high-reliability scene requirements of industrial Internet of Things, smart agriculture and the like are met.

Description

Technical Field

[0001] This invention relates to the field of gateways, specifically to a high-performance LoRaWAN gateway based on a Linux system and a data processing method thereon. Background Technology

[0002] Existing LoRaWAN gateway technologies typically employ an architecture that combines radio frequency drivers with a cloud protocol stack. The gateway device locally deploys the Semtech libloragw library and LoRa radio frequency module driver to receive LoRa signals from the air and forward the raw radio frequency data to the cloud server via the UDP protocol. The cloud then performs the parsing and frame processing of the LoRaWAN protocol.

[0003] However, this plan has the following shortcomings:

[0004] 1. The system is highly dependent on the stability of the network connection. If there is a network interruption or latency fluctuation (especially in a mobile communication environment), the LoRaWAN service will be interrupted immediately.

[0005] 2. The UDP-based transmission mechanism lacks reliable retransmission and flow control in unstable network environments such as 4G / 5G, which can easily lead to packet loss or out-of-order delivery, seriously affecting communication quality and system reliability.

[0006] To address the aforementioned issues, a high-performance LoRaWAN gateway and data processing method based on a Linux system are proposed. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology, adapt to practical needs, and provide a high-performance LoRaWAN gateway and data processing method based on the Linux system. This addresses the current system's high dependence on network connection stability, where LoRaWAN service will be immediately interrupted if network interruption or latency fluctuations occur (especially in mobile communication environments). Furthermore, the UDP-based transmission mechanism lacks reliable retransmission and flow control in unstable network environments such as 4G / 5G, easily causing data packet loss or out-of-order delivery, which seriously affects communication quality and system reliability.

[0008] To achieve the objectives of this invention, the technical solution adopted is as follows: A high-performance LoRaWAN gateway based on a Linux system is designed, the gateway comprising the following service modules:

[0009] The LoRa Package Forwarder service is used to initialize the LoRa RF module based on the Semtech libloragw library, set RF parameters such as center frequency, bandwidth, and spreading factor, and conduct bidirectional data interaction with the Chirpstack service through the local UDP protocol.

[0010] The Chirpstack service is used to implement the LoRaWAN protocol stack locally, process data frames such as JOIN, UpUnconfirm, DownUnconfirm, UpConfirm, and DownConfirm from Class A / B / C type devices, perform PHY header parsing, CRC check and AES-128-CMAC MIC integrity check, and write the verified LoRaWAN data frames to the Redis cache database.

[0011] The Redis cache database uses a Sorted Set structure to store the LoRaWAN data frames and sorts them by the received timestamp to support blocking reads and triggering subsequent processing.

[0012] HuaHongMgr is used to manage LoRaWAN nodes within Chirpstack via the gRPC interface, enabling the creation, deletion, modification, and querying of nodes; it also blocks the reading of newly written data frames from Redis and performs secondary validity verification on the data frames based on preset business rules (such as device whitelists and frame count anti-replay).

[0013] A local MySQL database is used to asynchronously and in batches insert LoRaWAN data frames that have been verified by the Encore MGR service into the storage in a transactional manner.

[0014] The cloud server (HuaHong Cloud) is used to receive and centrally process LoRaWAN data frames from multiple gateways via the MQTT protocol.

[0015] Preferably, the LoRa packet forwarding service and the Chirpstack service use a local UDP socket combined with a ring buffer shared memory (SHM) mechanism for efficient data transmission, so as to reduce memory copying overhead and reduce latency.

[0016] Preferably, when performing MIC verification on LoRaWAN data frames, the Chirpstack service uses the network session key (NwkSKey) combined with the AES-128-CMAC algorithm for integrity verification.

[0017] Preferably, the Redis cache database generates a globally unique identifier (UUID) for each data frame. The UUID is composed of the device EUI, the frame counter (FCnt), and the millisecond-level receive timestamp, and the ordered set is sorted and managed according to the UUID.

[0018] Preferably, after verifying and reading the data frames in the Redis cache, the Huahong MGR service further includes prioritizing and deduplicating multiple frames with the same DevAddr and FCnt based on RSSI and SNR metrics, and aggregating them according to a preset batch size or time window after removing redundant frames.

[0019] Preferably, the local MySQL database uses partitioned table technology to store data in partitions by device or month, and enables InnoDB compressed tables and write-ahead logs (WAL) to optimize write performance.

[0020] Preferably, after writing the verified LoRaWAN data frames into the MySQL database, the Huahong MGR service pushes them to the cloud server via the MQTT protocol in JSON format and QoS level 0, and caches the messages to be sent to the local file system in the form of a circular log file when the network is interrupted.

[0021] A method for processing LoRaWAN data frames is also provided, including the following steps:

[0022] S1, the LoRa packet forwarding service receives packets in real time based on the libroragw library and sends them to Chirpstack via the UDP communication protocol;

[0023] The S2 and Chirpstack services perform LoRa baseband demodulation, PHY header parsing, CRC removal, and AES-128-CMAC MIC verification on LoRa data packets. After the verification is successful, they attach UUID, RSSI / SNR, and timestamp to each frame of data and write it into a Redis ordered set.

[0024] S3 and Redis cache are sorted by timestamp and a blocking read is triggered, and Huahong MGR service is woken up and reads the new frame;

[0025] S4 and Huahong MGR services filter and deduplicate data frames based on DevAddr, FCnt, RSSI / SNR and ADRACKReq flags, and aggregate data frames by batch size or time window;

[0026] S5. Asynchronously batch insert the aggregated LoRaWAN data frames into the MySQL database in JSON or binary format;

[0027] S6. Push the stored messages to the cloud server via MQTT QoS 0. Cache the messages when the network is interrupted and retransmit them one by one in the original order after the network is restored.

[0028] S7. Repeat the steps.

[0029] Preferably, in step S2, CRC check is used to remove frames with transmission errors, and MIC check uses the AES-128-CMAC algorithm combined with the network session key for integrity verification.

[0030] Preferably, the local caching mechanism in step S6 is a circular log file format. Messages that fail to receive cloud ACK after exceeding a preset number of retransmissions are marked and written to an error queue for subsequent operation and maintenance analysis and alarms.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. This invention fully integrates the LoRaWAN protocol stack and libroragw RF driver locally on the gateway, realizing autonomous operation capability during network outages. Even in the event of network interruption or latency fluctuations, the gateway can still independently complete the reception, parsing, and node management of LoRaWAN data frames, thereby improving the reliability and availability of the system.

[0033] 2. This invention addresses issues such as UDP packet loss and out-of-order delivery in unstable network environments such as 4G / 5G by introducing retransmission confirmation and flow control at the local UDP layer, combined with Redis caching and other methods, which significantly reduces the risk of data loss and ensures high integrity of the LoRaWAN uplink.

[0034] 3. This invention integrates the LoRaWAN protocol stack and radio frequency driver, which were originally deployed separately on local and cloud platforms, into a single device. This simplifies the system architecture, reduces reliance on cloud computing resources, lowers deployment and maintenance costs, and facilitates widespread application in large-scale industrial IoT scenarios.

[0035] 4. This invention, through fully localized protocol stack processing, omits the cloud parsing step and minimizes network transmission latency, significantly shortening the entire process response time from RF reception and protocol parsing to storage / forwarding, thus meeting the needs of scenarios such as industrial control and smart agriculture with extremely high real-time requirements. Attached Figure Description

[0036] Figure 1 This is a system architecture diagram of the present invention;

[0037] Figure 2 This is a flowchart illustrating the LoRaWAN data frame processing of the present invention. Detailed Implementation

[0038] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0039] A high-performance LoRaWAN gateway and data processing method based on a Linux system, see [link to relevant documentation]. Figures 1 to 2 The gateway includes the following service modules:

[0040] The LoRa Package Forwarder service is used to initialize the LoRa RF module based on the Semtech libloragw library, set RF parameters such as center frequency, bandwidth, and spreading factor, and conduct bidirectional data interaction with the Chirpstack service through the local UDP protocol.

[0041] The Chirpstack service is used to implement the LoRaWAN protocol stack locally, process data frames such as JOIN, UpUnconfirm, DownUnconfirm, UpConfirm, and DownConfirm from Class A / B / C type devices, perform PHY header parsing, CRC check and AES-128-CMAC MIC integrity check, and write the verified LoRaWAN data frames to the Redis cache database.

[0042] The Redis cache database uses a Sorted Set structure to store LoRaWAN data frames and sorts them by the received timestamp to support blocking reads and triggering subsequent processing.

[0043] HuaHongMgr is used to manage LoRaWAN nodes within Chirpstack via the gRPC interface, enabling the creation, deletion, modification, and querying of nodes; it also blocks the reading of newly written data frames from Redis and performs secondary validity verification on the data frames based on preset business rules (such as device whitelists and frame count anti-replay).

[0044] A local MySQL database is used to asynchronously and in batches insert LoRaWAN data frames that have been verified by the Encore MGR service into the storage in a transactional manner.

[0045] The cloud server (HuaHong Cloud) is used to receive and centrally process LoRaWAN data frames from multiple gateways via the MQTT protocol.

[0046] Specifically, the LoRa packet forwarding service and the Chirpstack service use local UDP sockets combined with a ring buffer shared memory (SHM) mechanism for efficient data transmission, in order to reduce memory copying overhead and lower latency.

[0047] More specifically, when performing MIC verification on LoRaWAN data frames, the Chirpstack service uses the network session key (NwkSKey) combined with the AES-128-CMAC algorithm for integrity verification.

[0048] Furthermore, the Redis cache database generates a globally unique identifier (UUID) for each data frame. The UUID is composed of the device EUI, the frame counter (FCnt), and the millisecond-level receive timestamp, and the ordered set is sorted and managed according to the UUID.

[0049] Furthermore, after verifying and reading the data frames in the Redis cache, the Huahong MGR service also includes prioritizing and deduplicating multiple frames with the same DevAddr and FCnt based on RSSI and SNR metrics, and aggregating them according to a preset batch size or time window after removing redundant frames.

[0050] It is worth noting that the local MySQL database uses partitioned tables to store data by device or month, and InnoDB compressed tables and Write-Ahead Log (WAL) are enabled to optimize write performance.

[0051] It is worth noting that after writing the verified LoRaWAN data frames into the MySQL database, the Encore MGR service pushes them to the cloud server via the MQTT protocol in JSON format and QoS level 0. In the event of a network outage, the messages to be sent are cached in a circular log file to the local file system.

[0052] A method for processing LoRaWAN data frames is also provided, including the following steps:

[0053] S1, the LoRa packet forwarding service receives packets in real time based on the libroragw library and sends them to Chirpstack via the UDP communication protocol;

[0054] The S2 and Chirpstack services perform LoRa baseband demodulation, PHY header parsing, CRC removal, and AES-128-CMAC MIC verification on LoRa data packets. After the verification is successful, they attach UUID, RSSI / SNR, and timestamp to each frame of data and write it into a Redis ordered set.

[0055] S3 and Redis cache are sorted by timestamp and a blocking read is triggered, and Huahong MGR service is woken up and reads the new frame;

[0056] S4 and Huahong MGR services filter and deduplicate data frames based on DevAddr, FCnt, RSSI / SNR and ADRACKReq flags, and aggregate data frames by batch size or time window;

[0057] S5. Asynchronously batch insert the aggregated LoRaWAN data frames into the MySQL database in JSON or binary format;

[0058] S6. Push the stored messages to the cloud server via MQTT QoS 0. Cache the messages when the network is interrupted and retransmit them one by one in the original order after the network is restored.

[0059] S7. Repeat the steps.

[0060] It is worth mentioning that in step S2, CRC check is used to remove frames with transmission errors, and MIC check uses the AES-128-CMAC algorithm in combination with the network session key for integrity verification.

[0061] It is worth emphasizing that the local caching mechanism in step S6 is a circular log file format. Messages that have exceeded the preset number of retransmissions and have not received an ACK from the cloud are marked and written to the error queue for subsequent operation and maintenance analysis and alarms.

[0062] Example 1

[0063] Standard localized LoRaWAN gateway deployment

[0064] The gateway system of this invention was deployed on a single-board computer (1.2GHz, 1GB RAM) based on ARM Cortex-A53. The LoRa packet forwarding service calls the Semtech libloragw driver for the SX1301 RF front-end (center frequency 470.1MHz, bandwidth 125kHz, spreading factor 7) and communicates with the Chirpstack service via local UDP. The Chirpstack service parses the uplink data frames from the Class A device and writes the valid frames to Redis. The Huahong MGR service deduplicates and aggregates the frames according to the process described in claims 1–7, then asynchronously batches them into the local MySQL database and pushes them to the Huahong Cloud via the LTE module with MQTT QoS 0. In a test of continuously receiving 1000 frames / s, the end-to-end (RF to cloud) average latency was less than 120ms, and 100,000 messages could be cached without loss when the network was disconnected.

[0065] Example 2

[0066] High throughput optimization in scenarios with thousands of concurrent nodes

[0067] On the same hardware platform, the Redis cluster was deployed in a three-master, three-slave architecture, and MySQL was enabled with monthly partitioning and InnoDB compression. LoRa RF parameters were optimized to a bandwidth of 125kHz and a spreading factor of 9, and the number of concurrent Chirpstack threads was adjusted to 8 × the number of CPU cores. The Huahong MGR service used batch writing of 200 frames to MySQL for parallel transaction processing, achieving query performance for tens of millions of historical data. In actual testing with 1000 concurrent nodes (each node connecting at 1 second / frame), the system's average throughput remained above 800 frames / second, and the 99th percentile latency was below 200ms.

[0068] Example 3

[0069] Autonomous test of network outage under unstable 4G network environment

[0070] A gateway was deployed on-site, connected to the mobile network via a USB 4G card, and network jitter and intermittent disconnections were simulated: when the LTE link was interrupted for more than 5 seconds, the LoRa packet forwarding service and Chirpstack continued to receive and parse data, and the Huahong MGR service buffered messages in a circular log format; after the network was restored, all messages were automatically retransmitted without omission in the received sequence, and an exponential backoff retransmission strategy was executed. There was no single frame loss during the entire process, and the pending transmission buffer was cleared within 1 minute after the system was restored.

[0071] Example 4

[0072] Ethernet backhaul and remote operation and maintenance integration

[0073] A dual-network configuration is simultaneously set up on another gateway: the primary link is Ethernet, and the backup link is Wi-Fi. The gateway management module provides a gRPC-based web UI, allowing online modification of RF parameters, viewing of Redis queue depths and logs. It also integrates remote OTA functionality, enabling the distribution of new versions of Chirpstack and Huahong MGR service container images via MQTT and cold restarting of each module without requiring manual on-site maintenance. In operational testing, this solution achieved a mean time to recovery (MTTR) of less than 3 minutes.

[0074] Comparative Example 1

[0075] Cloud-based centralized resolution LoRaWAN gateway

[0076] The traditional architecture involves the LoRa RF module only receiving and transmitting LoRa data, with all UDP packets sent directly to the cloud for Chirpstack parsing; the gateway does not perform local protocol parsing or caching. In this mode, network jitter and high latency scenarios can lead to service unavailability, and uplink latency can reach 500ms–1s, failing to meet real-time requirements.

[0077] Comparative Example 2

[0078] Local UDP direct type casting, without buffering or retransmission mechanism

[0079] The gateway only deploys the libroragw forwarding service locally, continuously sending LoRa packets to the Chirpstack local process via UDP, but it does not use Redis caching, nor does it have local MQTT caching or file logs. This solution will experience permanent frame loss when the UDP packet loss rate is ≥2%, and it cannot retransmit historical data after the network recovers, resulting in poor reliability and integrity.

[0080] The specific comparison data is shown in the table below:

[0081]

[0082]

[0083] Summarize

[0084] Comparative Example 1

[0085] It relies entirely on the cloud for LoRaWAN protocol parsing and frame processing, lacking local caching and offline parsing capabilities; Comparative Example 2

[0086] It only forwards local UDP data to the local Chirpstack, without Redis caching, batch aggregation, or retransmission;

[0087] Comparison points:

[0088] End-to-end latency: The average latencies of Examples 1–4 are <120ms, <200ms (optimized for thousands of nodes), (not applicable to network outage testing), and (MTTR is a concern for operation and maintenance testing), respectively; Comparative Example 1 has a latency of 500ms–1s; Comparative Example 2 has an unquantifiable effective latency due to packet loss and out-of-order delivery.

[0089] Throughput capability: Example 2 shows a peak throughput of 800 frames / s; Example 1 is stable under a test of 1000 frames / s; Comparative Examples 1 / 2 show system unavailability or severe frame dropping under high concurrency.

[0090] Data integrity and reliability: Example 3 showed no frame loss in a 4G network outage environment, and all retransmissions were completed within 1 minute of network recovery; Comparative Examples 1 and 2 showed permanent frame loss and inability to retransmit during network jitter.

[0091] Operational Performance Time (MTTR): Example 4: Remote OTA and online configuration reduce MTTR to <3 minutes; Comparative Examples 1 / 2: No remote management and rapid recovery solutions, requiring on-site manual intervention for operation and maintenance.

[0092] in conclusion

[0093] The combined embodiments 1–4 significantly improve the system's real-time performance (latency <120ms), high throughput (800–1000 frames / s), reliability (no frame loss during network outages), and operational efficiency (MTTR <3min) through localized protocol parsing, Redis caching combined with batch aggregation, circular log retransmission, and modular remote management. In contrast, Comparative Example 1 suffers from high latency and interruption risk due to full cloud parsing, and Comparative Example 2 suffers from severe frame loss due to the lack of caching and retransmission mechanisms. Neither of these can meet the requirements of industrial-grade LoRaWAN gateways for low latency, high reliability, and maintainability.

[0094] In addition, all components designed in this invention are general standard parts or components known to those skilled in the art. Their structures and principles can be learned by those skilled in the art through technical manuals or conventional experimental methods. They can be fully implemented by those skilled in the art, so there is no need to elaborate. The content protected by this invention does not involve improvements to the internal structure and methods.

Claims

1. A high-performance LoRaWAN gateway based on a Linux system, characterized in that, The gateway includes the following service modules: The LoRa packet forwarding service is used to initialize the LoRa radio module based on the Semtech libloragw library, set the center frequency, bandwidth, and spreading factor radio parameters, and interact bidirectionally with the Chirpstack service via the local UDP protocol. The Chirpstack service is used to implement the LoRaWAN protocol stack locally, process JOIN, UpUnconfirm, DownUnconfirm, UpConfirm, and DownConfirm data frames of Class A / B / C type devices, perform PHY header parsing, CRC check and AES-128-CMAC MIC integrity check, and write the verified LoRaWAN data frames to the Redis cache database; The Redis cache database uses an ordered set structure to store the LoRaWAN data frames and sorts them by the received timestamp to support blocking reads and triggering subsequent processing. The Huahong MGR service is used to manage LoRaWAN nodes within Chirpstack via the gRPC interface, enabling node creation, deletion, modification, and querying; it also performs secondary validity verification on newly written data frames in Redis based on preset business rules. A local MySQL database is used to asynchronously and in batches insert LoRaWAN data frames that have been verified by the Encore MGR service into the storage in a transactional manner. The cloud server is used to receive and centrally process LoRaWAN data frames from multiple gateways via the MQTT protocol.

2. The gateway as described in claim 1, characterized in that, The LoRa packet forwarding service and the Chirpstack service use a local UDP socket combined with a ring buffer shared memory mechanism for efficient data transmission, thereby reducing memory copying overhead and lowering latency.

3. The gateway as described in claim 1, characterized in that, When performing MIC verification on LoRaWAN data frames, the Chirpstack service uses the network session key combined with the AES-128-CMAC algorithm for integrity verification.

4. The gateway as described in claim 1, characterized in that, The Redis cache database generates a globally unique identifier for each data frame. The UUID is composed of the device EUI, the frame counter, and the millisecond-level receiving timestamp, and the ordered set is sorted and managed according to the UUID.

5. The gateway as described in claim 1, characterized in that, After verifying and reading the data frames in the Redis cache, the Huahong MGR service also includes prioritizing and deduplicating multiple frames with the same DevAddr and FCnt based on RSSI and SNR metrics, and aggregating them according to a preset batch size or time window after removing redundant frames.

6. The gateway as described in claim 1, characterized in that, The local MySQL database uses partitioned tables to store data by device or month, and enables InnoDB compressed tables and write-ahead logs to optimize write performance.

7. The gateway as described in claim 1, characterized in that, After writing the verified LoRaWAN data frames into the MySQL database, the Huahong MGR service pushes them to the cloud server via the MQTT protocol in JSON format and QoS level 0. In the event of a network interruption, the messages to be sent are cached in a circular log file format to the local file system.

8. A method for processing LoRaWAN data frames, using the gateway described in any one of claims 1-7, characterized in that, Includes the following steps: S1, the LoRa packet forwarding service receives packets in real time based on the libroragw library and sends them to Chirpstack via the UDP communication protocol; The S2 and Chirpstack services perform LoRa baseband demodulation, PHY header parsing, CRC removal, and AES-128-CMAC MIC verification on LoRa data packets. After the verification is successful, they attach UUID, RSSI / SNR, and timestamp to each frame of data and write it into a Redis ordered set. S3 and Redis cache are sorted by timestamp and a blocking read is triggered, and Huahong MGR service is woken up and reads the new frame; S4 and Huahong MGR services filter and deduplicate data frames based on DevAddr, FCnt, RSSI / SNR and ADRACKReq flags, and aggregate data frames by batch size or time window; S5. Asynchronously batch insert the aggregated LoRaWAN data frames into the MySQL database in JSON or binary format; S6. Push the stored messages to the cloud server via MQTT QoS 0. Cache the messages when the network is interrupted and retransmit them one by one in the original order after the network is restored. S7. Repeat the steps.

9. The method as described in claim 8, characterized in that, In step S2, CRC check is used to eliminate frames with transmission errors, and MIC check uses the AES-128-CMAC algorithm combined with the network session key for integrity verification.

10. The method as described in claim 8, characterized in that, The local caching mechanism in step S6 is a circular log file format. Messages that fail to receive cloud ACK after exceeding the preset number of retransmissions are marked and written to the error queue for subsequent operation and maintenance analysis and alarms.

Images