A multi-protocol internet of things data cleaning and fusion method based on edge computing

Abstract

This invention relates to the field of Internet of Things (IoT) technology, specifically to a multi-protocol IoT data cleaning and fusion method based on edge computing. It addresses the shortcomings of IoT data acquisition terminals using hard-coded protocol-driven methods, which suffer from narrow protocol compatibility and poor scalability. It also addresses the drawbacks of centralized processing methods that directly transmit all raw data to the cloud, resulting in high network bandwidth consumption and heavy cloud computing and storage pressure. The method includes the following steps: S1: Constructing a multi-source heterogeneous protocol generalized access layer at the edge terminal, reserving various wired and wireless hardware interfaces to enable physical layer access for different types of sensors and controllers. This invention embeds lightweight data cleaning and fusion algorithms at the edge, completing preprocessing and feature extraction at the data source, significantly reducing invalid data transmission, effectively lowering network bandwidth consumption and cloud computing and storage pressure, and improving the overall system operating efficiency.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, specifically to a multi-protocol IoT data cleaning and fusion method based on edge computing. Background Technology

[0002] With the rapid development of the Internet of Things (IoT), the Industrial Internet, and smart cities, massive numbers of sensing devices are connected to the network, and heterogeneous terminals with multiple protocols, interfaces, and vendors have become typical characteristics of IoT systems. Currently, IoT data acquisition and transmission generally adopt a centralized cloud processing model, where various sensors and acquisition nodes directly upload raw data to the cloud platform for analysis, storage, and computation.

[0003] Existing IoT data acquisition terminals use hard-coded protocol-driven methods, which suffer from narrow protocol compatibility and poor scalability. Meanwhile, IoT data processing systems employ a centralized approach that directly transmits all raw data to the cloud, resulting in high network bandwidth consumption and significant cloud computing and storage pressure. Furthermore, edge data processing devices use simple threshold filtering, which struggles to achieve accurate cleaning and error correction of sensor data in complex environments, leading to poor uploaded data quality and low accuracy in subsequent analysis and decision-making. Therefore, these methods do not meet current requirements. To address this, we propose a multi-protocol IoT data cleaning and fusion method based on edge computing. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-protocol IoT data cleaning and fusion method based on edge computing, in order to solve the problems mentioned in the background art, which are caused by the use of hard-coded protocol-driven IoT data acquisition terminals, which have the disadvantages of narrow protocol adaptation range and poor scalability. At the same time, the IoT data processing system adopts a centralized processing method of directly transmitting the raw data to the cloud, which has the disadvantages of high network bandwidth consumption and high cloud computing and storage pressure. Furthermore, the edge data processing device uses a simple threshold filtering method, which is difficult to achieve accurate cleaning and error correction of sensor data in complex environments, resulting in poor uploaded data quality and low accuracy of subsequent analysis and decision-making.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multi-protocol IoT data cleaning and fusion method based on edge computing, comprising the following steps:

[0006] S1: Build a multi-source heterogeneous protocol generalized access layer at the edge terminal, and reserve a variety of wired and wireless hardware interfaces to realize physical layer access for different types of sensors and controllers;

[0007] S2: Develop a generalized protocol parsing middleware that dynamically loads protocol parsing scripts through object model description files distributed from the cloud, thereby completing the parsing and standardization conversion of heterogeneous data;

[0008] S3: Deploy a lightweight data cleaning algorithm model on the edge side to perform outlier detection, noise filtering, and missing value processing on the raw data;

[0009] S4: Extract and compress features from the cleaned data, and aggregate and prioritize the data according to preset rules;

[0010] S5: Uploads the processed standardized data to the cloud platform via a lightweight communication protocol, and receives control commands from the cloud to execute local linkage.

[0011] Preferably, the wired protocols supported by the multi-source heterogeneous protocol generalization access layer in step S1 include RS485, RS232, I2C, SPI and 4-20mA analog signals, and the wireless protocols supported include LoRa, NB-IoT, BLEMesh and WiFi.

[0012] Preferably, the generalized protocol parsing middleware described in step S2 adopts an object-oriented sensor abstraction layer (HAL) design. When a new sensor is connected, there is no need to recompile the underlying firmware; only the corresponding protocol parsing script needs to be sent to complete the driver loading.

[0013] Preferably, the standardization transformation in step S2 specifically involves: extracting the payload of the original message, stripping redundant checksums and headers, and mapping it to a unified JSON standard data structure within the system.

[0014] Preferably, the lightweight data cleaning algorithm model described in step S3 includes the Kalman filter algorithm and the moving average filter algorithm, wherein the state update formula of the Kalman filter algorithm is:

[0015]

[0016] In the formula, Let A be the state estimate at time k, and let A be the state transition matrix. Let B be the state estimate at time k-1, and let B be the control input matrix. For the control input at time k, This is process noise.

[0017] Preferably, the priority classification in step S4 specifically involves: performing low-frequency packaged uploading of steady-state data and performing high-priority immediate reporting of data with sudden changes.

[0018] Preferably, the lightweight communication protocol mentioned in step S5 is the MQTT protocol, which uses a publish or subscribe model to realize bidirectional communication between the terminal and the cloud platform.

[0019] Preferably, the local linkage mentioned in step S5 specifically means that the edge terminal directly executes control commands locally according to preset rules, and the response delay is controlled within 60ms.

[0020] An edge computing terminal for a multi-protocol IoT data cleaning and fusion method based on edge computing includes:

[0021] Main control module: Employs a microprocessor with floating-point arithmetic capabilities and abundant DMA channels;

[0022] Power management module: It adopts a power tree hierarchical power supply scheme, and the RF module and sensor peripherals are controlled by independent load switches;

[0023] Industrial protective enclosure: Made of UV-resistant materials, with a protection rating of IP65 or higher.

[0024] A cloud-edge collaborative IoT system based on a multi-protocol IoT data cleaning and fusion method using edge computing includes an edge computing terminal, a cloud platform, and a mobile application. The cloud platform adopts a microservice architecture and includes a data access gateway, a time-series database, and a rule engine.

[0025] Compared with the prior art, the beneficial effects of the present invention are:

[0026] 1. This invention achieves physical-level support and automatic protocol adaptation for various sensors by constructing a multi-source heterogeneous protocol generalization access layer and a dynamic protocol parsing middleware, thus completely breaking down data silos.

[0027] 2. This invention supports local execution of control logic on edge terminals, compressing response latency from seconds to milliseconds, and achieving true low-latency real-time linkage;

[0028] 3. This invention embeds a lightweight data cleaning and fusion algorithm at the edge, completing preprocessing and feature extraction at the data source, which greatly reduces invalid data transmission, effectively reduces network bandwidth usage and cloud computing and storage pressure, and improves the overall operating efficiency of the system. Attached Figure Description

[0029] Figure 1 This is an overall flowchart of the method of the present invention;

[0030] Figure 2 This is the overall architecture diagram of the cloud-edge collaborative IoT system of the present invention;

[0031] Figure 3 This is a typical application topology diagram of the present invention. Detailed Implementation

[0032] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0033] Please see Figures 1 to 3 The first embodiment provided by the present invention:

[0034] A multi-protocol IoT data cleaning and fusion method based on edge computing includes the following steps:

[0035] S1: Build a multi-source heterogeneous protocol generalized access layer at the edge terminal, and reserve a variety of wired and wireless hardware interfaces to realize physical layer access for different types of sensors and controllers;

[0036] S2: Develop a generalized protocol parsing middleware that dynamically loads protocol parsing scripts through object model description files distributed from the cloud, thereby completing the parsing and standardization conversion of heterogeneous data;

[0037] S3: Deploy a lightweight data cleaning algorithm model on the edge side to perform outlier detection, noise filtering, and missing value processing on the raw data;

[0038] S4: Extract and compress features from the cleaned data, and aggregate and prioritize the data according to preset rules;

[0039] S5: Uploads the processed standardized data to the cloud platform via a lightweight communication protocol, and receives control commands from the cloud to execute local linkage.

[0040] The multi-source heterogeneous protocol generalization access layer in step S1 supports wired protocols including RS485, RS232, I2C, SPI and 4-20mA analog signals, and wireless protocols including LoRa, NB-IoT, BLEMesh and WiFi.

[0041] The generalized protocol parsing middleware in step S2 adopts an object-oriented sensor abstraction layer (HAL) design. When a new sensor is connected, there is no need to recompile the underlying firmware; only the corresponding protocol parsing script needs to be sent to complete the driver loading.

[0042] The standardization transformation in step S2 specifically involves: extracting the payload of the original message, stripping redundant checksums and headers, and mapping it to a unified JSON standard data structure within the system.

[0043] The lightweight data cleaning algorithm model in step S3 includes the Kalman filter algorithm and the moving average filter algorithm. The state update formula of the Kalman filter algorithm is:

[0044]

[0045] In the formula, Let A be the state estimate at time k, and let A be the state transition matrix. Let B be the state estimate at time k-1, and let B be the control input matrix. For the control input at time k, This is process noise.

[0046] The priority classification in step S4 is as follows: low-frequency packaging and uploading is performed for steady-state data, and high-priority real-time reporting is performed for data with sudden changes.

[0047] The lightweight communication protocol in step S5 is the MQTT protocol, which uses a publish or subscribe model to achieve bidirectional communication between the terminal and the cloud platform.

[0048] The local linkage in step S5 is as follows: the edge terminal executes control commands directly on the local side according to preset rules, and the response delay is controlled within 60ms.

[0049] An edge computing terminal for a multi-protocol IoT data cleaning and fusion method based on edge computing includes:

[0050] Main control module: Employs a microprocessor with floating-point arithmetic capabilities and abundant DMA channels;

[0051] Power management module: It adopts a power tree hierarchical power supply scheme, and the RF module and sensor peripherals are controlled by independent load switches;

[0052] Industrial protective enclosure: Made of UV-resistant materials, with a protection rating of IP65 or higher.

[0053] A cloud-edge collaborative IoT system based on a multi-protocol IoT data cleaning and fusion method using edge computing includes an edge computing terminal, a cloud platform, and a mobile application. The cloud platform adopts a microservice architecture and includes a data access gateway, a time-series database, and a rule engine.

[0054] The second embodiment provided by the present invention:

[0055] Test gateways and 6-20 wireless data acquisition nodes were deployed at the following locations: Fengning Tea Base in Fengqing County, Lincang City (within line of sight, no obvious obstruction); Jiantou Yuncha Mountain Villa in Yuxi City (sparse vegetation, slight canopy obstruction, open slope); Xiao'aozi Coffee Plantation in Pu'er (moderate foliage obstruction); Shengyuan Fruit Tree Half-Red Plum Base in Suijiang County, Zhaotong (gentle hillside, undulating terrain + dense vegetation); and Tuanjiexin Apple Base in Xishan District, Kunming (crossing a ridge + severe physical + comprehensive vegetation obstruction). Test experiments were conducted on effective communication transmission distance, packet loss rate, and signal strength. The experimental records are as follows:

[0056] Table 1. Packet loss rate test of LoRa / NB-IoT wireless networking communication under complex terrain.

[0057]

[0058] Based on communication tests conducted in practical application scenarios of Yunnan's plateau-characteristic agriculture (such as tea gardens, coffee forests, orchards, and hilly terrain), the IoT intelligent data acquisition and aggregation terminal demonstrated excellent connection stability with a packet loss rate of only 0.90% in complex plateau mountain and hilly environments at a height of 650 meters. However, in an environment spanning a mountain ridge at a distance of 800 meters (with severe physical and vegetation obstruction), the packet loss rate increased to 18.40%. After replacing the antenna with a high-gain glass antenna, the packet loss rate decreased to 5.30%, demonstrating high connection stability even in harsh environments.

[0059] The third embodiment provided by the present invention:

[0060] The following test data was obtained by comparing the processing power, latency, and resource consumption of two gateways—a traditional direct-connect cloud architecture and an edge computing architecture—under different loads:

[0061] Table 2 Performance Comparison Test Data Between Edge Computing Architecture and Traditional Architecture

[0062]

[0063] In light-load scenarios (40 nodes), the edge computing terminal can complete local cleaning, filtering, and computation of 1200 data points per minute, achieving edge-end data preprocessing. This significantly reduces bandwidth and storage consumption. Traditional architectures directly upload massive amounts of raw sensor data to the cloud, which not only easily causes network congestion but also generates a large amount of invalid "dead data." With the edge computing architecture, the terminal only sends valid data that has been aggregated and anomaly detected to the cloud, reducing network bandwidth usage by more than 90% and cloud storage consumption by over 93%, effectively saving public network communication costs and cloud server resources.

[0064] Meanwhile, the traditional architecture of linkage control requires a long loop of "sensor → cloud processing → control command issuance", with a delay of up to several seconds under heavy load; the edge computing terminal of this project can directly execute the preset linkage control logic locally (such as triggering irrigation command immediately when soil drought is detected), which significantly shortens the control response delay, compressing the response delay from the second level to the millisecond level of about 50 milliseconds, and realizing true low-latency real-time linkage.

[0065] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. An edge computing-based multi-protocol Internet of Things data cleaning and fusion method, characterized in that, Includes the following steps: S1: Build a multi-source heterogeneous protocol generalized access layer at the edge terminal, and reserve a variety of wired and wireless hardware interfaces to realize physical layer access for different types of sensors and controllers; S2: Develop a generalized protocol parsing middleware that dynamically loads protocol parsing scripts through object model description files distributed from the cloud, thereby completing the parsing and standardization conversion of heterogeneous data; S3: Deploy a lightweight data cleaning algorithm model on the edge side to perform outlier detection, noise filtering, and missing value processing on the raw data; S4: Extract and compress features from the cleaned data, and aggregate and prioritize the data according to preset rules; S5: Uploads the processed standardized data to the cloud platform via a lightweight communication protocol, and receives control commands from the cloud to execute local linkage. 2.The edge computing based multi-protocol IoT data cleaning and fusion method according to claim 1, characterized in that, The multi-source heterogeneous protocol generalization access layer mentioned in step S1 supports wired protocols including RS485, RS232, I2C, SPI and 4-20mA analog signals, and wireless protocols including LoRa, NB-IoT, BLEMesh and WiFi. 3.The edge computing based multi-protocol IoT data cleaning and fusion method according to claim 1, characterized in that, The generalized protocol parsing middleware described in step S2 adopts an object-oriented sensor abstraction layer (HAL) design. When a new sensor is connected, there is no need to recompile the underlying firmware; only the corresponding protocol parsing script needs to be sent to complete the driver loading. 4.The edge computing based multi-protocol IoT data cleaning and fusion method according to claim 1, characterized in that, The standardization transformation described in step S2 specifically involves: extracting the payload of the original message, stripping redundant checksums and headers, and mapping it to a unified JSON standard data structure within the system. 5.The edge computing based multi-protocol IoT data cleaning and fusion method according to claim 1, characterized in that, The lightweight data cleaning algorithm model described in step S3 includes the Kalman filter algorithm and the moving average filter algorithm, wherein the state update formula of the Kalman filter algorithm is: wherein is the state estimate at time k, A is the state transition matrix, is the state estimate at time k - 1, B is the control input matrix, is the control input at time k, is the process noise.

6. The multi-protocol IoT data cleaning and fusion method based on edge computing according to claim 1, characterized in that, The priority classification mentioned in step S4 is as follows: low-frequency packaged uploading is performed for steady-state data, and high-priority real-time reporting is performed for data with sudden changes.

7. The multi-protocol IoT data cleaning and fusion method based on edge computing according to claim 1, characterized in that, The lightweight communication protocol mentioned in step S5 is the MQTT protocol, which uses a publish or subscribe model to achieve bidirectional communication between the terminal and the cloud platform.

8. The multi-protocol IoT data cleaning and fusion method based on edge computing according to claim 1, characterized in that, The local linkage mentioned in step S5 specifically means that the edge terminal directly executes control commands locally according to preset rules, and the response delay is controlled within 60ms.

9. An edge computing terminal for the multi-protocol IoT data cleaning and fusion method based on edge computing according to any one of claims 1-8, characterized in that, include: Main control module: Employs a microprocessor with floating-point arithmetic capabilities and abundant DMA channels; Power management module: It adopts a power tree hierarchical power supply scheme, and the RF module and sensor peripherals are controlled by independent load switches; Industrial protective housing: Made of UV-resistant materials, with a protection level of IP65 or higher.

10. A cloud-edge collaborative IoT system based on the multi-protocol IoT data cleaning and fusion method according to any one of claims 1-8, characterized in that, It includes edge computing terminals, cloud platforms, and mobile applications, with the cloud platform adopting a microservice architecture.

Images