Low-energy positioning data management system for large-scale sensor networks

Abstract

The application discloses a low-energy positioning data management system for a large-scale sensor network, and relates to the technical field of data processing.The application solves the technical problems of lacking full-link metadata management and automatic data life cycle scheduling, which leads to long-term occupation of storage resources by low-frequency data and low query efficiency of high-frequency data.The application removes positioning drift data through Kalman filtering, improves data effectiveness, adapts to BLE5.3, LoRaWAN and other protocols according to scenes, cooperates with differential coding and quantitative compression, reduces the amount of transmission data, reduces bandwidth occupancy, classifies high-frequency data according to real-time performance, matches high-performance / low-performance nodes, controls high-frequency core data access delay at the millisecond level, stores low-frequency data in near-line HDD-archiving object storage hierarchical storage, reduces unit storage cost, considers performance and economy, evaluates node performance through response delay+storage capacity two-dimensional normalization, and improves data storage reliability through a distributed storage combined with a copy mechanism.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a low-power positioning data management system for large-scale sensor networks. Background Technology

[0002] With the large-scale application of IoT technology in fields such as industrial monitoring, logistics tracking, and smart cities, the demand for location data collection and management is experiencing explosive growth.

[0003] According to patent application number 202110850825.2, a data processing method and a data storage management system are disclosed. After the client obtains the identifier of the data to be read, the client searches its own temporary cache data for related matching data with the data association to which the identifier belongs. If no related matching data is found, the target storage device is determined from multiple data storage devices. Then, a data acquisition request for the data association to which the identifier belongs is sent to the target storage device. This allows the target storage device to find the target data with the data association corresponding to the data acquisition request from its own storage data according to the data storage index list, and send all the found target data to the client. This enables fast access to multiple data with the same association and improves the high latency problem of accessing associated data.

[0004] Existing systems evaluate storage node performance using a single dimension, failing to dynamically adapt to data access loads. This leads to resource mismatch issues such as overloaded high-performance nodes and idle low-performance nodes. Furthermore, the lack of end-to-end metadata management and automated data lifecycle scheduling results in low-frequency data occupying storage resources for extended periods and low query efficiency for high-frequency data, making it difficult to balance the relationship between energy consumption, cost, and performance in data management. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a low-power positioning data management system for large-scale sensor networks, which solves the problems of lacking end-to-end metadata management and automated data lifecycle scheduling, resulting in low-frequency data occupying storage resources for a long time and low query efficiency for high-frequency data.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a low-power positioning data management system for large-scale sensor networks, comprising:

[0007] The positioning data acquisition module is used to collect positioning data and transmit it to the positioning data adaptive encoding module;

[0008] The adaptive encoding module for positioning data is used to filter positioning data to obtain preprocessed data, select a communication protocol from the scene-protocol mapping rule base in combination with the positioning scene, differentially encode and quantize the preprocessed data to obtain optimized positioning data, calculate the usage frequency by period, compare it with the threshold, classify the high-frequency / low-frequency usage data and transmit it to the corresponding module.

[0009] The hot data storage and analysis module is used for distributed storage of high-frequency data. It analyzes the response latency and capacity of storage nodes and normalizes and weights them to obtain a comprehensive index. It divides the nodes into high-performance and low-performance nodes and recalculates them periodically. It further classifies the high-frequency data according to its real-time performance and matches them with the corresponding nodes for storage. After deduplication and filtering, it builds a metadata index library and generates high-frequency storage management information.

[0010] The management information output module manages frequently used data based on frequently stored management information.

[0011] As a further aspect of the present invention, the positioning data acquisition module acquires positioning data in the following manner:

[0012] Instead of using a fixed sampling interval, the sampling interval is dynamically adjusted according to the stationary / moving state of the object being located or the needs of the scene. When stationary, the sampling interval is extended to the minute level. When moving, sampling is triggered by the accelerometer and the frequency is increased to the second level. After stationary, the frequency is automatically reduced.

[0013] As a further aspect of the present invention, the scene-protocol mapping rule base of the positioning data adaptive encoding module includes:

[0014] For indoor high-precision and low-dynamic scenarios, the BLE5.3 (AoA) protocol is preferred, as it meets the requirements of low power consumption and short-range high precision. For outdoor wide-area and high-dynamic scenarios, the LoRaWAN Class C protocol is preferred, as it meets the requirements of long-range coverage and low data rate. For emergency tracking scenarios, the NB-IoT (eDRX) protocol is preferred, as it meets the requirements of low latency and high reliability.

[0015] As a further aspect of the present invention, the differential encoding processing method of the positioning data adaptive encoding module is as follows:

[0016] Using the first data point after time-period aggregation as the baseline (x0, y0, t0), subsequent data only transmits the difference between the baseline value and the actual value. =x i -x0, =y i -y0, =t i -t0), which sets the quantization step size according to the scene's accuracy requirements, converts floating-point differences into integers.

[0017] As a further aspect of the present invention, the method of using data in high-frequency / low-frequency categories is as follows:

[0018] Using time T as a period, the number of times the optimized positioning data is used is obtained, and the corresponding usage frequency is calculated according to the formula usage frequency = number of times used / time T. Then, it is compared with the frequency threshold, and the specific value of the frequency threshold is set by the operator.

[0019] Data with a usage frequency greater than a frequency threshold is classified as high-frequency usage data, and data with a usage frequency less than a frequency threshold is classified as low-frequency usage data.

[0020] As a further aspect of the present invention, in the hot data storage and analysis module, the response delay normalization value = 1 - (node ​​delay / cluster maximum delay), and the remaining capacity normalization value = node remaining capacity / cluster average remaining capacity.

[0021] The corresponding comprehensive index is obtained by weighted summation. The comprehensive index is then compared with the corresponding judgment threshold. Storage nodes with comprehensive indexes greater than the judgment threshold are classified as high-performance nodes, while those with less comprehensive indexes are classified as low-performance nodes.

[0022] As a further aspect of the present invention, the real-time deduplication of the thermal data storage and analysis module is based on device ID + time window deduplication. Only the latest one of the three positioning data reported by the same device within 1 second is retained. When the coordinate deviation is <0.5 meters, the data is merged. The quality filtering is to remove abnormal data with a positioning reliability of <90%.

[0023] The metadata index includes a basic index, a feature index, and a related index. The basic index contains data ID, device ID, timestamp range, and storage node address. The feature index contains data quality level, update frequency, and last access time. The related index contains related business scenarios and replica distribution locations.

[0024] As a further aspect of the present invention, a cold data storage and analysis module is also included. This module is used to perform quality filtering and invalid data removal on low-frequency data. Based on the data time, it categorizes recent low-frequency data (within the last 3-6 months) and long-term low-frequency data (over 6 months), storing them respectively using a distributed HDD storage cluster and public / private cloud object storage. Migration rules are set so that when near-line layer data is stored for 6 months and the number of accesses during that period is ≤1, it is automatically migrated to the archive layer. A metadata index library is built to generate low-frequency storage management information.

[0025] As a further embodiment of the present invention, the distributed HDD storage cluster is CephRGW or GlusterFS, which is sharded according to the region to which the device belongs. Cross-regional redundancy is enabled for public cloud / private cloud object storage, and the quality filtering and invalid data removal methods of the cold data storage analysis module are consistent with the quality filtering and deduplication methods of the frequently used data in the hot data storage analysis module.

[0026] This invention provides a low-power positioning data management system for large-scale sensor networks. Compared with existing technologies, it has the following advantages:

[0027] This invention improves data validity by eliminating location-drifting data through Kalman filtering. It adapts to various scenarios using protocols such as BLE 5.3 and LoRaWAN, and incorporates differential coding and quantization compression to reduce the amount of data transmitted and lower bandwidth usage. High-frequency data is categorized based on real-time performance and matched with high-performance / low-performance nodes, with high-frequency core data access latency controlled to the millisecond level. Low-frequency data is stored in a tiered system of near-line HDD and archive object storage to reduce unit storage costs, balancing performance and economy. Node performance is evaluated using a dual-dimensional normalization of response latency and storage capacity, with node tiers dynamically updated every 5 minutes. High-performance nodes are temporarily downgraded when load surges to avoid overload. Distributed storage combined with a replication mechanism enhances data storage reliability. Attached Figure Description

[0028] Figure 1 This is a system block diagram of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] First Embodiment

[0031] Please see Figure 1 This application provides a low-power positioning data management system for large-scale sensor networks, including a positioning data acquisition module, a positioning data adaptive encoding module, a hot data storage and analysis module, a cold data storage and analysis module, and a management information output module, according to the appendix. Figure 1 It can be seen that the information between the above functional modules is transmitted in one direction.

[0032] The positioning data acquisition module is used to collect positioning data and transmit the collected positioning data to the positioning data adaptive encoding module. The acquisition method does not use a fixed sampling interval, but dynamically adjusts it according to the stationary / moving state of the positioning object or the needs of the scene. For example, when the object is stationary, the sampling interval is increased from the second level to the minute level, such as 1 minute / time, which greatly reduces the node wake-up frequency. When the object moves, sampling is triggered by the accelerometer, and the frequency is increased when moving, such as 1 second / time, and automatically reduced when stationary.

[0033] The adaptive encoding module for positioning data filters the acquired positioning data. Using a Kalman filter algorithm, it removes drifting data, such as jumps in data due to weak GPS signals, retaining only data with a confidence level greater than or equal to a threshold. Simultaneously, for multiple positioning results within the same time period, it calculates the average or median as a single valid data point, reducing the number of data points and obtaining preprocessed data. Next, it acquires the positioning scene corresponding to the preprocessed data and selects different communication protocols based on the positioning scene to generate transmission protocol information. Specifically, it establishes a scene-protocol mapping rule base, as follows:

[0034] Scene type Preferred Protocol Core adaptation points Indoor high precision + low dynamic range BLE5.3(AoA) Low power consumption + short distance and high precision, suitable for factory equipment positioning Outdoor wide area + high dynamic range LoRaWAN Class C Long-range coverage and low speed make it suitable for tracking logistics vehicles. Emergency tracking scenario NB-IoT (eDRX) Low latency and high reliability ensure priority transmission of alarm information.

[0035] Simultaneously, the preprocessed data and location data are compressed to reduce the number of bytes transmitted, and differential encoding is used for processing. Specifically, for time-series location data, the first data after time-segment aggregation is used as the baseline (x0, y0, t0), and subsequent data only transmits the difference between the baseline value and the baseline value. =x i -x0, =y i -y0, =t i -t0), set the quantization step size according to the scene accuracy requirements, convert the floating point difference to an integer to obtain the optimized positioning data, and at the same time, with time T as the period, obtain the number of times the optimized positioning data is used, and calculate the corresponding usage frequency according to the formula usage frequency = number of times used / time T. Then compare it with the frequency threshold, and the specific value of the frequency threshold is set by the operator.

[0036] Data with a usage frequency greater than a frequency threshold is classified as high-frequency usage data, and data with a usage frequency less than a frequency threshold is classified as low-frequency usage data. The classified data is then transmitted to the corresponding processing module.

[0037] The hot data storage and analysis module is used to manage the storage of frequently used data. It acquires and distributes this data, obtains all storage nodes, and analyzes their performance, specifically through response latency and storage capacity. Both are normalized: response latency: normalized value = 1 - (node ​​latency / maximum cluster latency); remaining capacity: normalized value = node remaining capacity / average cluster remaining capacity. Each indicator is mapped to the [0,1] range, with higher values ​​indicating better performance. A weighted sum is then used to obtain a comprehensive indicator. This comprehensive indicator is compared to a corresponding judgment threshold, set by the operator. Storage nodes with comprehensive indicators exceeding the threshold are classified as high-performance nodes, while those with lower indicators are classified as low-performance nodes. The comprehensive indicator is recalculated every 5 minutes. If the load rate of a high-performance node suddenly increases to over 80%, it is temporarily downgraded to a standard performance node to avoid overload.

[0038] Next, high-frequency usage data is acquired and then classified into high-frequency hot data and high-frequency cold data according to the usage scenarios corresponding to the high-frequency usage data. Specifically, based on the real-time requirements of the usage scenarios, data with high real-time requirements are classified into high-frequency hot data, and data with low real-time requirements are classified into high-frequency cold data. Then, high-frequency hot data is matched with high-performance nodes for storage, and high-frequency cold data is matched with low-performance nodes for storage, and storage matching signals are generated.

[0039] The generated storage matching signals are analyzed. For high-performance node matching, the distance between all high-performance nodes and the server is obtained, and nodes are selected based on the distance. At the same time, the corresponding high-frequency hot data is stored. Then, the high-frequency hot data is deduplicated and filtered in real time. Real-time deduplication is based on device ID + time window. For example, if three location data reports from the same device within 1 second, only the latest one is retained. Data with coordinate deviation <0.5 meters is merged. Quality filtering is used to remove abnormal data with location reliability <90%. A metadata index library is built to generate high-frequency storage management information and transmit it to the management information output module.

[0040] The specific metadata indexes include:

[0041] Basic index: Data ID, Device ID, Timestamp range, Storage node address;

[0042] Feature index: data quality level, update frequency, last access time;

[0043] Related indexes: related business scenarios and replica distribution locations.

[0044] The management information output module is used to manage the storage of frequently used data based on the acquired high-frequency storage management information.

[0045] Second Embodiment

[0046] As a second embodiment of the present invention, it is implemented based on the first embodiment, and the difference from the first embodiment is as follows:

[0047] The cold data storage and analysis module manages the storage of acquired low-frequency usage data. It performs quality filtering and removes invalid data from the low-frequency data, using the same processing methods as high-frequency usage data. The module then obtains the data time corresponding to the low-frequency usage data and stores it at different levels based on the data time. Specifically, for recent low-frequency data (represented by the last 3-6 months), it uses the near-line storage layer, employing a distributed HDD storage cluster such as CephRGW or GlusterFS, with sharding based on the device's region to reduce cross-region query latency. For older low-frequency data (represented by more than 6 months), it uses the archive storage layer, connecting to public / private cloud object storage and enabling cross-region redundancy to prevent data loss due to single-point failures.

[0048] At the same time, corresponding migration rules are set to realize the automatic migration of data between the near-line layer and the archive layer. Specifically, the data is judged based on both storage duration and access frequency. If the near-line layer data is stored for 6 months and the number of accesses during that period is ≤1, the migration to the archive layer is automatically triggered. At the same time, a metadata index library is established, low-frequency storage management information is generated, and it is transmitted to the management information output module.

[0049] The management information output module is used to manage the storage of low-frequency usage data based on the acquired low-frequency storage management information.

[0050] Third Embodiment

[0051] As a third embodiment of the present invention, the focus is on combining the implementation processes of the first and second embodiments.

[0052] Some of the data in the above formulas are numerical calculations with dimensions removed, and the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0053] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A low-power positioning data management system for large-scale sensor networks, characterized in that, include: The positioning data acquisition module is used to collect positioning data and transmit it to the positioning data adaptive encoding module; The location data adaptive encoding module is used to filter the location data to obtain preprocessed data, and select a communication protocol from the scene-protocol mapping rule base based on the location scene. The specific selection method is as follows: For indoor high-precision and low-dynamic scenarios, the BLE5.3 (AoA) protocol is selected to meet the requirements of low power consumption and short-range high precision. For outdoor wide-area and high-dynamic scenarios, the LoRaWAN Class C protocol is selected to meet the requirements of long-range coverage and low data rate. For emergency tracking scenarios, the NB-IoT (eDRX) protocol is selected to meet the requirements of low latency and high reliability. The preprocessed data is differentially encoded and quantized to obtain optimized positioning data. The usage frequency is calculated by period and compared with the threshold to distinguish between high-frequency and low-frequency usage data. The high-frequency usage data is then transferred to the hot data storage and analysis module and the low-frequency usage data is transferred to the cold data storage and analysis module. The hot data storage and analysis module is used for distributed storage of frequently used data. It analyzes the response latency and capacity of storage nodes and normalizes and weights them to obtain a comprehensive index. The normalized response latency value = 1 - (node ​​latency / maximum cluster latency) and the normalized remaining capacity value = node remaining capacity / average remaining cluster capacity. The corresponding comprehensive index is obtained by weighted summation. The comprehensive index is then compared with the corresponding judgment threshold. Storage nodes with comprehensive indexes greater than the judgment threshold are classified as high-performance nodes, while those with comprehensive indexes less than the judgment threshold are classified as low-performance nodes. The system divides data into high-performance and low-performance nodes and recalculates them periodically. High-frequency data is then classified into secondary categories based on real-time performance and matched with corresponding nodes for storage. After deduplication and filtering, a metadata index is built to generate high-frequency storage management information. The management information output module manages frequently used data based on frequently stored management information.

2. The low-power positioning data management system for large-scale sensor networks according to claim 1, characterized in that, The location data acquisition module acquires location data in the following way: Instead of using a fixed sampling interval, the sampling interval is dynamically adjusted according to the stationary / moving state of the object being located or the needs of the scene. When stationary, the sampling interval is extended to the minute level. When moving, sampling is triggered by the accelerometer and the frequency is increased to the second level. After stationary, the frequency is automatically reduced.

3. The low-power positioning data management system for large-scale sensor networks according to claim 1, characterized in that, The differential encoding processing method of the location data adaptive encoding module is as follows: Using the first data point after time-period aggregation as the baseline (x0, y0, t0), subsequent data only transmits the difference between the baseline value and the actual value. =x i -x0, =y i -y0, =t i -t0), which sets the quantization step size according to the scene's accuracy requirements, converts floating-point differences into integers.

4. The low-power positioning data management system for large-scale sensor networks according to claim 1, characterized in that, The method of using data in high-frequency / low-frequency categories is as follows: Using time T as a period, the number of times the optimized positioning data is used is obtained, and the corresponding usage frequency is calculated according to the formula usage frequency = number of times used / time T. Then, it is compared with the frequency threshold, and the specific value of the frequency threshold is set by the operator. Data with a usage frequency greater than a frequency threshold is classified as high-frequency usage data, and data with a usage frequency less than a frequency threshold is classified as low-frequency usage data.

5. The low-power positioning data management system for large-scale sensor networks according to claim 1, characterized in that, The real-time deduplication of the hot data storage and analysis module is based on device ID + time window. Only the latest one of the three positioning data reported by the same device within 1 second is retained. Data with coordinate deviation <0.5 meters is merged. The quality filter removes abnormal data with a positioning reliability of <90%. The metadata index includes a basic index, a feature index, and a related index. The basic index contains data ID, device ID, timestamp range, and storage node address. The feature index contains data quality level, update frequency, and last access time. The related index contains related business scenarios and replica distribution locations.

6. The low-power positioning data management system for large-scale sensor networks according to claim 1, characterized in that, It also includes a cold data storage and analysis module, which is used to filter low-frequency data and remove invalid data. Based on the data time, it divides recent low-frequency data (3-6 months) and long-term low-frequency data (more than 6 months) into distributed HDD storage clusters and public / private cloud object storage, respectively. Set migration rules: when near-line layer data is stored for 6 months and accessed ≤ 1 time during that period, it will be automatically migrated to the archive layer, and a metadata index will be built to generate low-frequency storage management information.

7. The low-power positioning data management system for large-scale sensor networks according to claim 6, characterized in that, The distributed HDD storage cluster is CephRGW or GlusterFS, with sharded storage based on the region to which the device belongs. Cross-regional redundancy is enabled for public / private cloud object storage, and the quality filtering and invalid data removal methods of the cold data storage analysis module are consistent with the quality filtering and deduplication methods of the frequently used data in the hot data storage analysis module.

Images