Computer data storage method and system based on Internet of Things

Abstract

The invention discloses a computer data storage method and system based on the Internet of Things, and relates to the technical field of the Internet of Things, the computer data storage system comprises an equipment access module, a state perception module, a decision scheduling module and the like, heterogeneous data is received, then subjected to protocol adaptation standardization and stored at the edge and the cloud end in a hierarchical mode according to a dynamic strategy, data synchronization is achieved, and query is supported; the method has the advantages that edge node storage resources, cloud loads and network states are perceived in real time through a reinforcement learning model of the decision scheduling module, an edge-cloud storage quota proportion and a synchronization strategy are dynamically generated, elastic allocation of the resources is achieved, and the resource utilization rate is improved. And the synchronization control module executes real-time synchronization according to the strategy when the network is stable, so that the problems of unbalanced static resource allocation in edge-cloud collaborative storage and low data synchronization efficiency in a weak network environment are effectively solved, and finally the resource utilization rate and synchronization reliability of data storage of the Internet of Things are improved.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, specifically to a computer data storage method and system based on IoT. Background Technology

[0002] As the core driving force of IoT applications, data's efficient storage and management are the key foundations for supporting the implementation of IoT technology. In smart home scenarios, the real-time status data generated by smart home devices needs to be reliably stored to ensure remote control and scene linkage. In the industrial manufacturing field, the storage of high-frequency operating data of production equipment provides data support for production optimization and fault diagnosis. In the construction of smart cities, the continuous storage of data such as traffic flow and environmental monitoring is a prerequisite for achieving refined urban management.

[0003] The applicant discovered through a search that Chinese patents disclose "a method, device and medium for unified scheduling of resources in a heterogeneous cloud environment" (publication number CN120086002A) and "a multi-protocol data communication method and system for intelligent buildings" (publication number CN120321315A). The applicant found that although "CN120086002A" proposes cloud resource scheduling, it still adopts a preset quota strategy and does not solve the dynamic synchronization problem under weak network conditions. Furthermore, the protocol adaptation of "CN120321315A" is only for the communication layer and does not involve the protocol-storage instruction mapping of the storage system. Therefore, we propose a computer data storage method and system based on the Internet of Things. Summary of the Invention

[0004] The purpose of this invention is to provide a computer data storage method and system based on the Internet of Things.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a computer data storage system based on the Internet of Things, comprising a data storage system, wherein the data storage system includes:

[0006] Device access module: Used to receive various types of data (including time-series data, video stream data, and device status data) sent by heterogeneous IoT devices (including sensors, smart terminals, and industrial controllers), and to perform protocol parsing and format standardization processing on the data;

[0007] Status awareness module: Collects system operating status parameters in real time, including edge node storage resource utilization, cloud storage node load, network bandwidth quality and data transmission latency, and sends the collected status parameters to the decision scheduling module;

[0008] Decision scheduling module: Receives status parameters sent by the status awareness module, dynamically generates resource allocation strategy and data synchronization strategy based on reinforcement learning model. The resource allocation strategy is used to determine the storage quota ratio between edge nodes and the cloud, and the data synchronization strategy is used to select real-time synchronization and asynchronous synchronization mode. The generated strategies are sent to the edge storage module, cloud storage module and synchronization control module respectively.

[0009] Edge storage module: Receives standardized data sent by the device access module, stores the data according to the resource allocation strategy of the decision scheduling module, and establishes a data index locally;

[0010] Cloud storage module: Receives some data synchronized from the edge storage module (based on the strategy of the decision scheduling module) and high-priority data directly transmitted from the device access module, and performs distributed storage and redundant backup;

[0011] Synchronization control module: Receives synchronization strategy from decision scheduling module, controls data transmission between edge storage module and cloud storage module, switches to pre-synchronization and local caching mode when network is unstable, and performs data retransmission after network recovery;

[0012] Protocol adaptation module: It has a built-in unified protocol abstraction layer, which converts the heterogeneous protocols of IoT devices (including MQTT, CoAP, OPCUA) into common operation commands within the system, enabling data interaction adaptation between the device access module, edge storage module, and cloud storage module;

[0013] Data query module: Receives user query requests, retrieves data from the edge storage module and cloud storage module according to the data index, and returns standardized query results.

[0014] As a further aspect of the present invention: the state parameters collected by the state sensing module specifically include:

[0015] Edge node parameters: remaining storage capacity of edge node (in GB, real-time refresh rate is 1 time / second), CPU utilization of edge node (percentage, sampling period is 500ms) and memory usage of edge node (percentage, statistics by process).

[0016] Cloud node parameters: total load of cloud storage nodes (number of concurrent write requests, statistical granularity is 1 minute), storage utilization rate of each cloud partition (statistics based on data shards) and cloud node response time (time elapsed from receiving a request to completing storage, in ms).

[0017] Network parameters: Real-time network bandwidth (uplink / downlink rate, in Mbps, collected every 2 seconds), data transmission packet loss rate (the percentage of lost data packets in the past 100 seconds), network fluctuation coefficient (the standard deviation of bandwidth in the past 30 seconds).

[0018] Data characteristic parameters: data type identifier (time series data is labeled "T", video stream data is labeled "V" and device status data is labeled "S"), data generation frequency (number of data entries per unit time, such as 10 time series data entries generated per second by the sensor) and data priority label (high priority is labeled "P1", medium priority is labeled "P2", and low priority is labeled "P3").

[0019] As a further aspect of the present invention: the reinforcement learning model of the decision scheduling module includes a state space, an action space, and a reward function, wherein the state space is a set of real-time data volume of the device access module, network bandwidth of the state perception module, and storage resource parameters; the action space is a set of edge-cloud storage quota adjustment values ​​and synchronization cycle adjustment values; the reward function is calculated based on data transmission latency and storage resource utilization and is used to optimize the decision strategy.

[0020] The reward function calculation formula for the decision scheduling module is as follows:

[0021] ;

[0022] in: This is the reward value, ranging from 0 to 1. A higher value indicates a better strategy. This represents the storage resource utilization rate of edge nodes, ranging from 0 to 1. The higher the utilization rate, the lower the score for this item. This is the data synchronization latency coefficient, ranging from 0 to 1. The higher the latency, the lower the score for this item. This represents the balance of cloud storage resource utilization, ranging from 0 to 1. The smaller the standard deviation of utilization for each partition, the higher the score for this item. , This represents the standard deviation of the usage rate of each partition in the cloud, with a value ranging from 0 to 1. , and These are the weighting coefficients, and + + =1.

[0023] As a further aspect of the present invention: the synchronization control module receives network bandwidth data from the status perception module in real time. When the network bandwidth is detected to be lower than the preset threshold for three consecutive times, it is determined to be in an unstable network state. The pre-synchronization mode is automatically started, and the P1 level data for the next 5 minutes is compressed and cached to the edge temporary storage area. After the network is restored, it is retransmitted in batches of 1MB / s and verified by CRC32. Real-time synchronization is performed when the network is stable, and batch synchronization is performed every 30 seconds when the network fluctuates.

[0024] As a further aspect of the present invention: the unified protocol abstraction layer of the protocol adaptation module includes:

[0025] Protocol identification unit: Identifies the protocol type by the handshake message characteristics during the device initialization phase. Specifically, this includes: parsing the header fields of the handshake message (such as the fixed header byte 0x10 of the MQTT protocol and the version field 0x01 of the CoAP protocol), extracting the protocol identifier string (such as the “OPCUABinary” identifier of the OPCUA protocol), and matching it with a preset protocol feature code library (which stores the feature values ​​of 20 mainstream IoT protocols).

[0026] Instruction Conversion Unit: Constructs a protocol-instruction mapping table to convert operation instructions of different protocols into system-wide universal opcodes. Specifically, the "PUBLISH" instruction of the MQTT protocol, the "PUT" instruction of the CoAP protocol, and the "Write" service of the OPCUA protocol are uniformly mapped to "STORE_0x01" (data write operation); the "SUBSCRIBE" instruction of the MQTT protocol and the "GET" instruction of the CoAP protocol are uniformly mapped to "QUERY_0x02" (data query operation).

[0027] Format adaptation unit: standardizes data formats from different protocols into the system's internal format;

[0028] Extended compatibility unit: Supports dynamic protocol expansion. When an unknown protocol device is connected, it automatically extracts instruction format features and updates the protocol feature code library by parsing the first 10 data frames sent by the device, and can complete the adaptation to the new protocol without restarting the system.

[0029] The instruction conversion unit of the protocol adaptation module adopts a dynamic mapping algorithm, and the conversion accuracy calculation formula is as follows:

[0030] ;

[0031] in, This represents the conversion accuracy, ranging from 0 to 1. A higher value indicates a better conversion result. The number of instructions that are successfully converted and executed without errors. This represents the total number of instructions per unit of time, with a statistical period of 1 minute. This represents the protocol difference coefficient, ranging from 0 to 1. The syntax difference coefficient between MQTT and OPCUA is 0.3, and the syntax difference coefficient between CoAP and MQTT is 0.1. This represents the data complexity coefficient, ranging from 0 to 1. For video stream data, it is 0.8; for time-series data, it is 0.3; and for text state data, it is 0.1. This is a correction factor (fixed value 0.2) used to compensate for the impact of protocol differences and data complexity on conversion accuracy, ensuring a conversion accuracy of ≥95% in complex scenarios.

[0032] As a further aspect of the present invention: the local data index of the edge storage module adopts a time-series-spatial hybrid structure: the time-series index layer constructs a B+ tree with timestamps, each leaf node corresponds to a 5-minute data block, and the query response is ≤100ms; the spatial index layer stores latitude and longitude coordinates based on an R tree, and supports geographic range queries by associating video clip paths; the index update unit is updated in real time through a transaction mechanism and stored in the high-speed SSD area.

[0033] As a further solution of the present invention: the cloud storage module adopts a dynamic sharding and redundancy strategy, shards according to the dual dimensions of "device ID hash + timestamp", and distributes it to 3 nodes through consistent hashing. There are 3 replicas of P1 level data, the primary node + 2 off-site backup nodes, 2 replicas of P2 level data, 1 replica of P3 level data, and 6+3 fault tolerance coding. Cold data (no access for 7 days) is migrated to object storage, and queries are seamless.

[0034] As a further aspect of the present invention: the data query module supports multi-dimensional queries, including: querying data for a specified time period by device ID, querying a list of associated devices by data type, and querying video stream data by geographical location range, and appending a data source identifier (edge ​​storage / cloud storage) to the query results.

[0035] In addition, this application also provides a computer data storage method based on the Internet of Things, the data storage method comprising the following steps:

[0036] Step 1: Data Access and Adaptation: Receive various types of data sent by heterogeneous IoT devices, convert heterogeneous protocols into general operation instructions through a unified protocol abstraction layer, and standardize the data format.

[0037] Step 2, State Awareness and Policy Generation: Real-time collection of state parameters such as edge node storage resources, cloud load, network bandwidth and data characteristics, and dynamic generation of edge-cloud storage quota policies and data synchronization policies based on reinforcement learning models;

[0038] Step 3, Tiered Storage: Based on the generated storage quota policy, standardized data is stored separately on edge nodes (to establish local data indexes) and cloud nodes (for distributed storage and redundant backup), with high-priority data being directly transferred to cloud storage;

[0039] Step 4: Dynamic Synchronization Control: Control data transmission according to the synchronization strategy and network status. Real-time or batch synchronization is performed when the network is stable. Pre-synchronization and local caching are started when the network is unstable. After the network recovers, cached data is retransmitted through verification.

[0040] Step 5, Data Query Response: Receive user query requests, retrieve data from edge or cloud storage based on the data index, and return standardized query results with additional data source identifiers.

[0041] Compared with the prior art, the beneficial effects of the present invention by adopting the above technical solution are as follows:

[0042] 1. This invention uses a reinforcement learning model in the decision scheduling module to perceive edge node storage resources, cloud load, and network status in real time, dynamically generating edge-cloud storage quota ratios and synchronization strategies to achieve elastic resource allocation. The synchronization control module performs real-time synchronization when the network is stable and initiates a pre-synchronization mechanism to cache key data in advance when the network is unstable. After the network recovers, data integrity is ensured through verification and retransmission. Combined with the edge-cloud hierarchical storage architecture to adapt to data priority differences, this method effectively solves the problems of static resource allocation imbalance and low data synchronization efficiency in weak network environments in edge-cloud collaborative storage, ultimately improving the resource utilization and synchronization reliability of IoT data storage.

[0043] 2. This invention utilizes a unified protocol abstraction layer in the protocol adaptation module. A protocol identification unit parses handshake message features to match protocol types, while an instruction conversion unit uses a dynamic mapping algorithm to convert heterogeneous protocol operation instructions into universal operation codes. A format adaptation unit unifies data formats and retains metadata, and an extension compatibility unit automatically extracts new protocol features to achieve dynamic adaptation. This approach effectively solves the problems of interaction barriers caused by protocol differences among heterogeneous IoT devices and the complexity of adapting new protocols. Ultimately, it improves the system's compatibility with multiple types of devices and the flexibility of protocol extension, ensuring smooth storage and interaction of data from different devices. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the reinforcement learning strategy generation process of the decision scheduling module in this embodiment of the invention;

[0045] Figure 2 This is a schematic diagram of the system flow in an embodiment of the present invention. Detailed Implementation

[0046] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0047] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0048] Please see the appendix Figure 1 -Appendix Figure 2 This invention discloses a computer data storage system based on the Internet of Things, comprising a data storage system, which includes:

[0049] Device access module: Used to receive various types of data sent by heterogeneous IoT devices, and to perform protocol parsing and format standardization processing on the data;

[0050] Status awareness module: collects system operating status parameters in real time, including edge node storage resource utilization, cloud storage node load, network bandwidth quality, and data transmission latency;

[0051] Decision scheduling module: Based on reinforcement learning model, it dynamically generates resource allocation strategy and data synchronization strategy. The resource allocation strategy is used to determine the storage quota ratio between edge nodes and the cloud, and the data synchronization strategy is used to select real-time synchronization and asynchronous synchronization mode.

[0052] Edge storage module: Receives standardized data sent by the device access module, stores the data according to the resource allocation strategy of the decision scheduling module, and establishes a data index locally;

[0053] Cloud storage module: Receives some data synchronized from the edge storage module and high-priority data directly transmitted from the device access module, and performs distributed storage and redundant backup;

[0054] Synchronization control module: Receives synchronization strategy from decision scheduling module, controls data transmission between edge storage module and cloud storage module, switches to pre-synchronization and local caching mode when network is unstable, and performs data retransmission after network recovery;

[0055] Protocol adaptation module: It has a built-in unified protocol abstraction layer, which converts the heterogeneous protocols of IoT devices into common operation commands within the system, enabling data interaction adaptation between the device access module, edge storage module, and cloud storage module;

[0056] Data query module: Receives user query requests, retrieves data from the edge storage module and cloud storage module according to the data index, and returns standardized query results.

[0057] In one embodiment of the present invention: the state parameters collected by the state sensing module specifically include:

[0058] Edge node parameters: remaining storage capacity of the edge node, CPU utilization of the edge node, and memory usage of the edge node;

[0059] Cloud node parameters: total load of cloud storage nodes, storage utilization of each cloud partition, and response time of cloud nodes;

[0060] Network parameters: real-time network bandwidth, data transmission packet loss rate, network fluctuation coefficient;

[0061] Data characteristic parameters: data type identifier, data generation frequency, and data priority label.

[0062] In one embodiment of the present invention: the reinforcement learning model of the decision scheduling module includes a state space, an action space, and a reward function, wherein the state space is a set of real-time data volume of the device access module, network bandwidth of the state perception module, and storage resource parameters; the action space is a set of edge-cloud storage quota adjustment values ​​and synchronization cycle adjustment values; the reward function is calculated based on data transmission latency and storage resource utilization and is used to optimize the decision strategy;

[0063] The reward function calculation formula for the decision scheduling module is as follows:

[0064] ;

[0065] in: The reward value ranges from 0 to 1. The storage resource utilization rate for edge nodes, ranging from 0 to 1. This is the data synchronization delay coefficient, ranging from 0 to 1. The cloud storage resource utilization balance is measured in increments of 0-1. , The standard deviation of the utilization rate of each partition in the cloud. , and These are the weighting coefficients, and + + =1.

[0066] In one embodiment of the present invention: the synchronization control module receives network bandwidth data from the status sensing module in real time. When the network bandwidth is detected to be lower than the preset threshold for three consecutive times, it is determined to be in an unstable network state. The pre-synchronization mode is automatically started, and the P1 level data for the next 5 minutes is compressed and cached to the edge temporary storage area. After the network is restored, it is retransmitted in batches of 1MB / s and verified by CRC32. Real-time synchronization is performed when the network is stable, and batch synchronization is performed every 30 seconds when the network fluctuates.

[0067] In one embodiment of the present invention: the unified protocol abstraction layer of the protocol adaptation module includes:

[0068] Protocol identification unit: Identifies the protocol type based on the characteristics of the handshake messages during the device initialization phase;

[0069] Instruction conversion unit: Constructs a protocol-instruction mapping table to convert operation instructions of different protocols into system-internal universal opcodes;

[0070] Format adaptation unit: standardizes data formats from different protocols into the system's internal format;

[0071] Extended compatibility unit: By parsing the first 10 data frames sent by the device, it automatically extracts instruction format features and updates the protocol feature code library;

[0072] The instruction conversion unit of the protocol adaptation module adopts a dynamic mapping algorithm, and the conversion accuracy calculation formula is as follows:

[0073] ;

[0074] in, For conversion accuracy, the range is 0-1. The number of instructions that are successfully converted and executed without errors. This represents the total number of instructions per unit of time. This is the protocol difference coefficient, ranging from 0 to 1. is the data complexity coefficient.

[0075] In one embodiment of the present invention: the local data index of the edge storage module adopts a time-series-spatial hybrid structure: the time-series index layer constructs a B+ tree with timestamps, each leaf node corresponds to a 5-minute data block, and the query response is ≤100ms; the spatial index layer stores latitude and longitude coordinates based on an R tree, and supports geographic range queries by associating video clip paths; the index update unit is updated in real time through a transaction mechanism and stored in the high-speed SSD area.

[0076] In one embodiment of the present invention: the cloud storage module adopts a dynamic sharding and redundancy strategy, sharding according to the dual dimensions of "device ID hash + timestamp", and distributing it to 3 nodes through consistent hashing. There are 3 replicas of P1 level data, the primary node + 2 off-site backup nodes, 2 replicas of P2 level data, 1 replica of P3 level data, and 6+3 fault tolerance coding. Cold data is migrated to object storage, and queries are seamless.

[0077] In one embodiment of the present invention: the data query module supports multi-dimensional queries, including: querying data for a specified time period by device ID, querying a list of associated devices by data type, and querying video stream data by geographical location range, and appending a data source identifier to the query results.

[0078] Example 1, please refer to the appendix. Figure 1 -Appendix Figure 2 Industrial IoT production workshop data storage scenario:

[0079] An IoT data storage system is being implemented in an automotive parts manufacturing workshop. The workshop is equipped with 100 industrial robots, 500 temperature / vibration sensors, 30 high-definition surveillance cameras, and 10 PLC controllers. The system needs to store equipment operation data (sampling frequency 10Hz), sensor time-series data (sampling frequency 1Hz), surveillance video streams (25fps, 1080P resolution), and equipment fault alarm data in real time. The data storage latency must be ≤100ms, fault data must be stored with high reliability, and cross-device data association queries must be supported.

[0080] System module implementation

[0081] Device access module

[0082] Access device types: industrial robots (OPCUA protocol), temperature sensors (MQTT protocol), surveillance cameras (video streams encapsulated by RTSP protocol), and PLC controllers (Modbus protocol).

[0083] Data processing: Parsing OPCUA binary data frames, MQTT JSON messages, and RTSP H.265 video streams, and uniformly converting them into a standardized structure containing "device ID + timestamp + data type + value" (e.g., temperature data format: { :" ", timestamp:1620000000000, type: "T", value: 35.2});

[0084] State awareness module

[0085] Edge node parameters: workshop edge gateway (2 units, 8GB memory, 1TB SSD) remaining storage capacity (refreshed in real time, current values ​​are 600GB and 550GB respectively), CPU utilization (sampling period 500ms, current average 35%), memory usage (28%).

[0086] Cloud node parameters: Total load of cloud storage cluster (10 nodes) (800 concurrent write requests / minute), storage utilization of each partition (standard deviation σ=5%), response time (30ms).

[0087] Network parameters: workshop LAN bandwidth (20Mbps uplink, 50Mbps downlink, collected every 2 seconds), packet loss rate (0.5%), network fluctuation coefficient (bandwidth standard deviation of the past 30 seconds = 2Mbps);

[0088] Data characteristic parameters: Temperature sensor data (marked "T", generation frequency 1 record / second, priority "P2"), robot fault data (marked "S", generation frequency random, priority "P1"), monitoring video (marked "V", generation frequency 25 frames / second, priority "P2");

[0089] Decision scheduling module

[0090] Reinforcement learning model state space: device access data volume (50MB / s), edge remaining storage (600GB / 550GB), cloud load (800 times / minute), network bandwidth (20Mbps);

[0091] Action space: Edge storage quota adjusted to 60% (robot fault data, real-time temperature data), cloud quota 40% (historical temperature data, video stream), synchronization cycle (P1 level data real-time synchronization, P2 level data batch synchronization every 30 seconds).

[0092] Reward function calculation: using industrial scenario weights ( =0.2, =0.5, =0.3), current =40% =0.1 (delay 10ms) =1−0.05=0.95, then =0.2×(1−0.4)+0.5×(1−0.1)+0.3×0.95=0.12+0.45+0.285=0.855 (strategy is optimal);

[0093] Edge storage module

[0094] Storage allocation: 60% quota (600GB edge gateway storage for robot fault data and temperature data for the past 24 hours);

[0095] Index structure: Temporal index (B+ tree, with one leaf node corresponding to each 5-minute data block, stored in locations such as / edge / temp / 20240601 / 08:00-08:05), Spatial index (R tree, storing the camera's latitude and longitude (30.1234°, 120.5678°), associated video clip path / edge / video / cam_01 / 20240601 / 08:00-08:05.mp4);

[0096] Cloud storage module

[0097] Storage allocation: 40% quota (receiving historical temperature data synchronized from the edge, video stream data, and directly transmitted P1-level fault data);

[0098] Sharding strategy: Shard by "Device ID hash (first 8 digits) + timestamp (hour level)", such as robot The data sharding at time 2024060108 is hash( )-08 (size 800MB), allocated to 3 cloud nodes (primary node + 2 off-site backup nodes);

[0099] Redundancy strategy: P1 level fault data is backed up with 3 copies, P2 level temperature data is backed up with 2 copies, and video stream data uses 6+3 fault-tolerant coding;

[0100] Synchronous control module

[0101] When the network is stable (bandwidth ≥ 200kbps, packet loss rate < 5%): P1 level fault data is synchronized in real time (latency from edge to cloud < 10ms), and P2 level data is synchronized in batches every 30 seconds (50MB each time).

[0102] When the network is unstable (due to interference from workshop equipment, the bandwidth drops to 150kbps and is below the threshold for three consecutive collections): start the pre-synchronization mode, compress and cache the P1 level data of the next 5 minutes (about 20MB, LZ77 compression rate 60%) to the edge temporary storage area (100GB capacity).

[0103] After the network is restored (bandwidth recovers to 250kbps and lasts for 10 seconds): retransmit the cached data in batches of 1MB, each batch is checked by CRC32 (checksum such as 0x12345678), and delete the temporary cache after the retransmission is completed;

[0104] Protocol adaptation module

[0105] Protocol identification: Parsing the "OPCUABinary" identifier in the OPCUA handshake message and the MQTT fixed header byte 0x10, matching it with the feature code library, the identification accuracy is 100%.

[0106] Command translation: OPCUA's "Write" service and MQTT's "PUBLISH" command are mapped to "STORE-0x01", and OPCUA's "Read" service and MQTT's "SUBSCRIBE" command are mapped to "QUERY-0x02".

[0107] Conversion accuracy: Out of 1000 instructions per minute, 980 are successfully converted. The protocol difference coefficient D = 0.3 (OPCUA vs. MQTT), and the data complexity T = 0.3 (time-series data), then:

[0108] =(980 / 1000)×(1−0.2×0.3×0.3)=0.98×0.982=0.962(96.2%≥95%)

[0109] Data query module

[0110] Supports querying by device ID. "Query the fault data from 08:00 to 09:00 on June 1, 2024 (returns 10 records in edge storage), and query the video clips of the corresponding cameras by geographical range "30.12°-30.13° North Latitude" (returns 3 video paths in cloud storage). The results are appended with the source identifier "edge" or "cloud".

[0111] Example 2, please refer to the appendix. Figure 1 -Appendix Figure 2 Smart home multi-device data storage scenarios

[0112] A smart home system encompasses 10 smart devices (smart door locks, temperature and humidity sensors, curtain motors, cameras, robot vacuums, etc.). It needs to store device status data (such as door lock opening and closing records), environmental sensor data (sampling frequency 0.5Hz), and camera monitoring video (15fps, 720P). It is required to support real-time data access for device linkage (such as automatically turning on lights after the door lock is opened) and adapt to home Wi-Fi network fluctuation scenarios.

[0113] II. Specific Implementation of System Modules

[0114] Device access module

[0115] Access device protocols: smart door lock (CoAP protocol), temperature and humidity sensor (MQTT protocol), camera (HTTP protocol), robot vacuum cleaner (Bluetooth to MQTT gateway);

[0116] Data standardization: Convert CoAP's TLV format and MQTT's JSON format to a unified Protocol Buffers format, preserving metadata (such as door lock data: { :" ", timestamp:1620000000000, type: "S", value: "open"});

[0117] State awareness module

[0118] Edge node parameters: Home gateway (1GB memory, 64GB eMMC) remaining storage capacity (40GB, 1 refresh / second), CPU utilization (15%, 500ms sampling), memory usage (20%).

[0119] Cloud node parameters: cloud load (30 concurrent writes / minute), partition utilization standard deviation σ=3%, response time (50ms).

[0120] Network parameters: Home Wi-Fi bandwidth (5Mbps uplink, 20Mbps downlink, collected every 2 seconds), packet loss rate (2%), network fluctuation coefficient (5Mbps);

[0121] Data characteristics: temperature and humidity data (labeled "T", 0.5 records / second, "P2"), door lock status (labeled "S", randomly generated, "P1"), camera video (labeled "V", 15fps, "P3");

[0122] Decision scheduling module

[0123] Reinforcement learning model: using smart home scene weights ( =0.4, =0.3, =0.3), current =30% =0.2 (delay 20ms) =1−0.03=0.97, then R=0.4×(1−0.3)+0.3×(1−0.2)+0.3×0.97=0.28+0.24+0.291=0.811 (strategy is better).

[0124] Strategy: 70% quota at the edge (door lock status, temperature and humidity data in the last 12 hours), 30% quota at the cloud (historical temperature and humidity, video data), real-time synchronization of P1 level data, and batch synchronization of P2 / P3 level data every 5 minutes;

[0125] Edge storage module

[0126] Indexes: Time-series index (B+ tree, 5-minute data blocks, 80ms query response), Spatial index (R tree storing camera location (30.1234°, 120.5678°), associated with local video clips);

[0127] Cloud storage module

[0128] Sharding: Sharding is done by "Device ID hash + timestamp (days)", such as sharding temperature and humidity sensor data as hash( )-20240601 (size 50MB), P1 level data 3 copies, P3 level video 1 copy + 6 + 3 encoding;

[0129] Synchronous control module

[0130] When the network is stable (Wi-Fi bandwidth ≥ 500kbps): door lock status is synchronized in real time, and temperature and humidity data are synchronized in batches every 5 minutes;

[0131] When the network is unstable (bandwidth drops to 300kbps for 3 consecutive times): pre-synchronize door lock data (about 5KB) for the next 5 minutes to the edge temporary storage area (6GB), and re-transmit it in batches of 1MB after recovery (actually only 1 batch).

[0132] Protocol adaptation module

[0133] Protocol Conversion: CoAP's "PUT" command and MQTT's "PUBLISH" command are mapped to "STORE-0x01". Conversion accuracy calculation:

[0134] =(990 / 1000)×(1−0.2×0.1×0.3)=0.99×0.994=0.984(98.4%);

[0135] Data query module

[0136] Supported search: Press " "Query the on / off records of the past 24 hours (returns 8 data entries from edge storage), and query the temperature and humidity trends of the past 7 days by pressing "type=T" (returns statistical data from cloud storage). The results are accompanied by source identifiers."

[0137] Based on the above embodiments, it can be concluded that the storage quota and synchronization strategy are dynamically generated by the reinforcement learning model of the decision scheduling module. The synchronization control module performs real-time synchronization when the network is stable and pre-synchronization and verification retransmission when the network is unstable. Combined with hierarchical storage to adapt data priority, the protocol adaptation module achieves dynamic adaptation of heterogeneous protocols through identification, conversion, adaptation and extension units. This effectively solves the problems of static resource imbalance between edge and cloud, inefficient synchronization in weak network, interaction barriers between heterogeneous devices and complex adaptation of new protocols. With the help of module collaboration, the storage resource utilization, synchronization reliability and device compatibility and protocol extension flexibility are improved.

[0138] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Any variations and modifications can be made by those skilled in the art without departing from the spirit and scope of the invention. Therefore, any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention, without departing from the scope of the invention, fall within the protection scope defined by the claims of the present invention.

Claims

1. A computer data storage system based on the Internet of Things, comprising a data storage system, characterized in that, The data storage system includes: Device access module: Used to receive various types of data sent by heterogeneous IoT devices, and to perform protocol parsing and format standardization processing on the data; Status awareness module: collects system operating status parameters in real time, including edge node storage resource utilization, cloud storage node load, network bandwidth quality, and data transmission latency; Decision scheduling module: Based on reinforcement learning model, it dynamically generates resource allocation strategy and data synchronization strategy. The resource allocation strategy is used to determine the storage quota ratio between edge nodes and the cloud, and the data synchronization strategy is used to select real-time synchronization and asynchronous synchronization mode. Edge storage module: Receives standardized data sent by the device access module, stores the data according to the resource allocation strategy of the decision scheduling module, and establishes a data index locally; Cloud storage module: Receives some data synchronized from the edge storage module and high-priority data directly transmitted from the device access module, and performs distributed storage and redundant backup; Synchronization control module: Receives synchronization strategy from decision scheduling module, controls data transmission between edge storage module and cloud storage module, switches to pre-synchronization and local caching mode when network is unstable, and performs data retransmission after network recovery; Protocol adaptation module: It has a built-in unified protocol abstraction layer, which converts the heterogeneous protocols of IoT devices into common operation commands within the system, enabling data interaction adaptation between the device access module, edge storage module, and cloud storage module; Data query module: Receives user query requests, retrieves data from the edge storage module and cloud storage module according to the data index, and returns standardized query results.

2. The computer data storage system based on the Internet of Things according to claim 1, characterized in that, The state parameters collected by the state awareness module specifically include: Edge node parameters: remaining storage capacity of the edge node, CPU utilization of the edge node, and memory usage of the edge node; Cloud node parameters: total load of cloud storage nodes, storage utilization of each cloud partition, and response time of cloud nodes; Network parameters: real-time network bandwidth, data transmission packet loss rate, network fluctuation coefficient; Data characteristic parameters: data type identifier, data generation frequency, and data priority label.

3. The computer data storage system based on the Internet of Things according to claim 1, characterized in that, The reinforcement learning model of the decision scheduling module includes a state space, an action space, and a reward function. The state space is a set of real-time data volume of the device access module, network bandwidth of the state awareness module, and storage resource parameters. The action space is a set of edge-cloud storage quota adjustment values ​​and synchronization cycle adjustment values. The reward function is calculated based on data transmission latency and storage resource utilization and is used to optimize the decision strategy. The reward function calculation formula for the decision scheduling module is as follows: ； in: The reward value ranges from 0 to 1. The storage resource utilization rate for edge nodes, ranging from 0 to 1. This is the data synchronization delay coefficient, ranging from 0 to 1. The cloud storage resource utilization balance is measured in increments of 0-1. , The standard deviation of the utilization rate of each partition in the cloud. , and These are the weighting coefficients, and + + =1.

4. The computer data storage system based on the Internet of Things according to claim 1, characterized in that: The synchronization control module receives network bandwidth data from the status perception module in real time. When the network bandwidth is detected to be lower than the preset threshold for three consecutive times, it is determined to be in an unstable network state. The pre-synchronization mode is automatically started, and the P1 level data for the next 5 minutes is compressed and cached to the edge temporary storage area. After the network is restored, it is retransmitted in batches of 1MB and verified by CRC32. Real-time synchronization is performed when the network is stable, and batch synchronization is performed every 30 seconds when the network fluctuates.

5. A computer data storage system based on the Internet of Things according to claim 1, characterized in that, The unified protocol abstraction layer of the protocol adaptation module includes: Protocol identification unit: Identifies the protocol type based on the characteristics of the handshake messages during the device initialization phase; Instruction conversion unit: Constructs a protocol-instruction mapping table to convert operation instructions of different protocols into system-internal universal opcodes; Format adaptation unit: standardizes data formats from different protocols into the system's internal format; Extended compatibility unit: By parsing the first 10 data frames sent by the device, it automatically extracts instruction format features and updates the protocol feature code library; The instruction conversion unit of the protocol adaptation module adopts a dynamic mapping algorithm, and the conversion accuracy calculation formula is as follows: ； in, For conversion accuracy, the range is 0-1. The number of instructions that are successfully converted and executed without errors. This represents the total number of instructions per unit of time. This is the protocol difference coefficient, ranging from 0 to 1. is the data complexity coefficient.

6. A computer data storage system based on the Internet of Things according to claim 1, characterized in that: The local data index of the edge storage module adopts a time-series-spatial hybrid structure: the time-series index layer constructs a B+ tree with timestamps, each leaf node corresponds to a 5-minute data block, and the query response is ≤100ms; the spatial index layer stores latitude and longitude coordinates based on an R tree, and supports geographic range queries by associating video clip paths; the index update unit is updated in real time through a transaction mechanism and stored in the high-speed SSD area.

7. A computer data storage system based on the Internet of Things according to claim 1, characterized in that: The cloud storage module adopts a dynamic sharding and redundancy strategy, sharding by two dimensions: 'device ID hash + timestamp', and distributing the data to 3 nodes through consistent hashing. There are 3 replicas of P1 level data, with the primary node and 2 off-site backup nodes, 2 replicas of P2 level data, 1 replica of P3 level data, and 6+3 fault-tolerant coding. Cold data is migrated to object storage, and queries are seamless.

8. A computer data storage system based on the Internet of Things according to claim 1, characterized in that: The data query module supports multi-dimensional queries, including: querying data for a specified time period by device ID, querying a list of associated devices by data type, and querying video stream data by geographical location range, and attaching data source identifiers to the query results.

9. A computer data storage method based on the Internet of Things applicable to the data storage system of any one of claims 1-8, characterized in that, The data storage method includes the following steps: Step 1: Data Access and Adaptation: Receive various types of data sent by heterogeneous IoT devices, convert heterogeneous protocols into general operation instructions through a unified protocol abstraction layer, and standardize the data format. Step 2, State Awareness and Policy Generation: Real-time collection of edge node storage resources, cloud load, network bandwidth and data characteristic state parameters, and dynamic generation of edge-cloud storage quota policies and data synchronization policies based on reinforcement learning models; Step 3, Tiered Storage: Based on the generated storage quota policy, standardized data is stored on edge nodes and cloud nodes respectively, with high-priority data being directly transmitted to cloud storage; Step 4: Dynamic Synchronization Control: Control data transmission according to the synchronization strategy and network status. Real-time and batch synchronization are performed when the network is stable. Pre-synchronization and local caching are started when the network is unstable. After the network recovers, cached data is retransmitted through verification. Step 5, Data Query Response: Receive user query requests, retrieve data from edge and cloud storage based on data indexes, and return standardized query results with additional data source identifiers.

Images