Unified storage and query method and system for multi-element sensor data

Abstract

The invention relates to a unified storage and query method and system for multi-element sensor data, which can access sensor equipment at any time without manual intervention. Sensor groups can be set and modified at any time, and data receiving and query are not affected; a unified storage method is adopted for different sensor data, and transverse expansion can be easily achieved; and the powerful storage capability of Hadoop is fully utilized, and meanwhile, flexible data query is realized.

Description

technical field [0001] The invention relates to the technical field of information processing, in particular to a method and system for unified storage and query of multi-sensor data. Background technique [0002] At present, the development of the Internet of Things has matured. Various sensing devices that need to be used in the construction of judicial administrative application systems, such as prison management sensing devices, executive correction bracelets, etc., can be divided into field types and remote sensing types from the perspective of observation and measurement standards. etc., from the observation platform mobility, can be divided into fixed sensors and mobile sensors. So many types of sensors have different data types, data composition, data transmission frequency, and data accuracy. The traditional method is to set up a separate data storage table in the relational database according to different sensor types. Store a row alone. Over time, these massively...

AI Technical Summary

Problems solved by technology

These methods all use the cache to receive real-time data. After the cache is full, they are written to the storage in batches at one time. The advantage is that the data writing performance can be improved, but there are several problems. 1. Failures such as system downtime and power failure will inevitably lead to cache 2. There is no solution for sensor diversity; 3. It cannot meet the application scenarios of real-time writing and real-time query, which is unacceptable in the emergency command system environment where time is racing against time; the application publication number is CN 104750814 A "Multi-Sensor-Based Multivariate Heterogeneous Data Flow Automatic Storage Method" realizes the multi-sensor data writing method, using the sensor format information table, monitoring date table, abnormal character table and repair function table to store corresponding information respectively

Before receiving heterogeneous streaming data, this method needs to establish sensor format information tables according to the sensor data format, which cannot flexibly deal with the diversification of multiple sensors. At the same time, using various repair functions to repair data will lose the authenticity of the data, resulting in Analysis scenarios similar to sensor stability analysis cannot be realized, and usually these abnormal data should be processed by data users according to actual needs; the application publication number is CN 110737807 A "A Data Storage Method, Driving Recorder, Server and storage media" uses the index feature of the EXT file system to use the date of data collection as the first-level index and the time of data collection as the second-level index. This method is highly dependent on the EXT file system and is not suitable for the HDFS file system. Data query based on the time range needs to perform multiple file scans, and the way of storing indexes and data blocks together reduces the efficiency of index scans; the application publication number is CN 110647497 A "a high-performance file storage based on HDFS In "And Management System", when querying the corresponding files, it is necessary to scan the index files sequentially, which cannot quickly locate the target, and the efficiency is not high. It is not suitable for frequently read application scenarios. In addition, for frequently read files, copy them to The local optimization method has little meaning for the scene with high randomness, and it is not suitable for the application scene of the present invention

Images