Data storage method and device based on multi-level cache and intelligent distribution and medium

Abstract

The invention provides a data storage method and device based on multi-level cache and intelligent distribution and a medium, and belongs to the technical field of data storage. The data storage method comprises the following steps: constructing a multi-stage cache architecture between a data acquisition end and a persistent storage end of Internet of Things data, wherein the multi-stage cache architecture at least comprises a local memory cache, a distributed cache and a database cache; according to the service priority, the change frequency and the storage cost of the Internet of Things data, dynamically routing the Internet of Things data to a corresponding cache level in a multi-level cache architecture through an intelligent distribution engine; performing data aggregation processing on the Internet of Things data, executing a timing batch writing strategy, and writing the Internet of Things data into a persistent storage end; and by adopting a data fragmentation and read-write separation strategy, dispersing and storing the data to a plurality of database instances or nodes of a persistent storage end so as to realize load balancing. According to the method and the device, the persistence of the mass sensing data with fidelity, load reduction and long period can be realized.

Description

Technical Field

[0001] This application belongs to the field of data storage technology, specifically relating to a data storage method, device, and medium based on multi-level caching and intelligent traffic distribution. Background Technology

[0002] The Industrial Internet of Things (IIoT) has undergone a rapid transformation from "device networking" to "data-driven operation and maintenance." According to the "2023 China Industrial Internet of Things White Paper," the amount of sensor data generated daily by a single large industrial device has reached 500GB to 2TB. This data needs to be collected within milliseconds, stored within seconds, and analyzed within minutes. To cope with this high-frequency write pressure, the industry has generally adopted a two- or three-layer architecture of "sensor → Redis → time-series / relational database." In this architecture, Redis acts as a centralized cache at the edge, using asynchronous threads to batch-write data to disk. With its high single-threaded throughput and support for data structures such as lists and streams, Redis has become a de facto "data bus." Downstream time-series databases (such as InfluxDB, TimescaleDB, or IoTDB) are responsible for compressed data storage, while relational databases (such as PostgreSQL or MySQL) are used to store configurations and analysis results.

[0003] Leading domestic and international platforms, such as Siemens MindSphere, GE Predix, Huawei ROMA, and Alibaba IoTStack, are all adopting the aforementioned architecture and have made intensive patent deployments. For example, Siemens' patent CN110097159A uses a "Redis+Kafka" dual-buffering mechanism to solve the peak impact problem; Huawei's patent CN114512728A proposes a method of "time-series library sharding + memory page writing" to reduce disk random I / O; and Alibaba Cloud's patent CN113984137A uses "data hot and cold tiering" to compress historical tablespaces. These solutions perform well in high-bandwidth, large-memory cloud environments, but they expose some common problems in local area network offline PHM scenarios.

[0004] First, there's the issue of single-level cache overload. When Redis reaches its memory limit and triggers LRU eviction or RDB / AOF persistence, the instantaneous CPU utilization can exceed 90%, causing the SDK's transmission failure rate to rise to 5%-15%, thus creating a time gap in the training set. GE Predix's field deployment report indicates that critical fault evolution segments of 3-5 minutes can be lost within 2 hours, which is enough to cause the PHM model to miss early, weak fault features.

[0005] Secondly, there's the issue of persistent write amplification. To alleviate the pressure on Redis, some solutions attempt to directly "de-cachise" and write data directly to the time-series database. This results in the database continuously operating under high concurrency, small message volume, and / or random writes. InfluxDB's official benchmark tests show that with random writes exceeding 100,000 points per second, disk IOPS write latency can be amplified by 3-10 times. Compaction threads compete for resources with compaction, increasing query latency from milliseconds to seconds. In offline server environments, due to the lack of elastic scaling capabilities in the cloud, once the disk is full, only earlier data can be repeatedly deleted, leading to truncation of historical baseline data and a decrease in PHM training accuracy.

[0006] To address these shortcomings, relevant companies have proposed several improvement measures. For example, Siemens, in its 2022 patent DE102022001A1, proposed adding a "local NVMe ring buffer" before Redis to temporarily store data for 30 minutes during a failure, and then re-fill it after network recovery. Huawei, in its 2023 patent CN117788300A, uses "edge streaming quality scoring" to mark low signal-to-noise ratio segments as dirty data and roll them back. However, these solutions still have significant shortcomings: 1. The ring buffer only solves the single point of failure of "network intermittent interruptions" and does not alleviate the CPU / memory bottleneck of Redis itself; 2. Streaming scoring relies on FPGA hardware acceleration, which is costly and can only filter noise; 3. All data still needs to pass through a single Redis instance, failing to achieve horizontal distribution of write load, and disk lifespan issues remain.

[0007] Therefore, in a PHM environment, how to simultaneously achieve "fidelity, load reduction, and long cycle" persistence of massive sensor data remains an industry gap that urgently needs to be addressed. Summary of the Invention

[0008] This application aims to provide an IoT data storage optimization method, device, and medium based on multi-level caching and intelligent traffic distribution. By introducing multi-level caching and intelligent traffic distribution mechanisms, it effectively solves the problem of persisting massive amounts of data in the data persistence process encountered by existing technologies, and achieves zero packet loss, load balancing, and long-term traceability of data, thereby providing a more reliable and efficient solution for data management and analysis of industrial IoT.

[0009] According to a first aspect of this application, this application provides an IoT data storage optimization method based on multi-level caching and intelligent traffic splitting, including: A multi-level caching architecture is built between the data acquisition end and the persistent storage end of IoT data. The multi-level caching architecture includes at least local memory cache, distributed cache and database cache. Based on the data characteristics and / or business needs of IoT data, the intelligent routing engine dynamically routes IoT data to the corresponding cache level in the multi-level cache architecture. The IoT data is aggregated and processed, and a timed batch write strategy is executed to write the data to persistent storage. By employing a data sharding and read-write separation strategy, data is distributed and stored across multiple database instances or nodes on the persistent storage end to achieve load balancing.

[0010] Preferably, in the above-mentioned IoT data storage optimization method, a multi-level caching architecture is constructed between the IoT data acquisition end and the persistent storage end. The multi-level caching architecture includes at least local memory caching, distributed caching, and database caching, including: In a multi-level caching architecture, a local memory cache is used as the first-level cache to store frequently accessed short-term data. In a multi-level caching architecture, a distributed cache cluster is used as a second-level cache to distribute data across multiple nodes, providing high availability and load balancing for data caching. In the multi-level caching architecture, a database cache is used as the third-level cache, which includes a time-series database and a relational database for persistent storage of IoT data.

[0011] Preferably, the above-mentioned IoT data storage optimization method, based on the data characteristics and / or business needs of IoT data, dynamically routes IoT data to the corresponding cache level in the multi-level cache architecture through an intelligent routing engine, including: The intelligent routing engine makes data routing decisions based on at least one of the following factors: business priority, waveform signal data or data change frequency, data storage cost and access latency requirements.

[0012] Preferably, the above-mentioned IoT data storage optimization method, which performs data aggregation processing on IoT data and executes a timed batch write strategy to write the data to persistent storage, includes: Downsampling and compression processing is performed on IoT data in waveform signal form; Based on the data requirements of the project management system, perform in-memory aggregation calculations on the feature value data; Based on the dynamic write frequency, the aggregated feature values ​​are written to persistent storage in batches; the dynamic write frequency is dynamically adjusted according to business needs.

[0013] Preferably, the above-mentioned IoT data storage optimization method employs a data sharding and read-write separation strategy, distributing data across multiple database instances or nodes on the persistent storage end, including: Hash sharding based on device ID, time range, or data tag; By using parallel processing, the hash-sharded data is distributed and stored across multiple database instances to achieve parallel data processing and load balancing.

[0014] Preferably, the above-mentioned IoT data storage optimization method employs a data sharding and read-write separation strategy to distribute data across multiple database instances or nodes on a persistent storage end, including: A master-slave replication architecture is used to achieve read-write separation of IoT data; the master-slave replication architecture includes a master database and a slave database, with the master database handling write operations and the slave database handling read operations. Combine caching systems to cache frequently accessed data in IoT data to improve read operation performance.

[0015] Preferably, the above-mentioned IoT data storage optimization method employs a data sharding and read-write separation strategy to distribute data across multiple database instances or nodes on a persistent storage end, including: IoT data is compressed and distributed across multiple nodes through distributed caching in a multi-level caching architecture; The compressed IoT data is written to a time-series database or relational database in a multi-level caching architecture through a locking mechanism.

[0016] Preferably, the above-mentioned method for optimizing IoT data storage employs a locking mechanism to prevent read / write conflicts when writing IoT data, including: A write lock is acquired before IoT data is written using a pre-defined configuration command in the database. The write lock is used to ensure that only one write operation occurs at a time. During reading, check the write lock status. If the write lock is held, wait or return an error message to avoid reading inconsistent data. Once the write operation is complete, release the write lock to allow read operations.

[0017] According to a second aspect of this application, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the IoT data storage optimization method based on multi-level caching and intelligent load balancing provided by any of the above technical solutions.

[0018] According to a third aspect of this application, this application also provides a computer storage medium storing a computer program thereon, which, when executed, implements the IoT data storage optimization method based on multi-level caching and intelligent traffic distribution provided by any of the above technical solutions.

[0019] The technical solution of this application has at least the following technical effects: The IoT data storage optimization method based on multi-level caching and intelligent traffic distribution provided in this application effectively solves the challenges encountered by existing solutions in the data persistence process by introducing multi-level caching and intelligent traffic distribution mechanisms. It achieves zero packet loss, load balancing and long-term traceability of data, making it possible to persist massive amounts of sensor data with "high fidelity, load reduction and long cycle", thereby providing a more reliable and efficient solution for data management and analysis of industrial IoT. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating an IoT data storage optimization method based on multi-level caching and intelligent traffic splitting, provided for an embodiment of this application; Figure 2 This application provides a schematic diagram of the data transmission architecture of a PHM system. Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To more clearly illustrate the overall concept of this application, a detailed explanation is provided below with reference to the accompanying drawings.

[0022] Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below. It should be noted that, unless otherwise specified, the embodiments of this application and the features thereof can be combined with each other.

[0023] In this application, unless otherwise expressly specified and limited, the terms "above" and "below" the second feature can refer to direct contact between the first and second features, or indirect contact between the first and second features through an intermediate medium. In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples.

[0024] The existing technology has the following drawbacks: Currently, most mainstream edge data persistence solutions in the industry are based on a "single-level cache + single-database optimization" architecture. However, these solutions have not yet been able to simultaneously solve the three core problems of "zero packet loss, zero hotspots, and long persistence" in the data persistence process. For example, Siemens MindSphere Industrial Edge Hub adopts a dual-buffering mechanism of "Redis 5.0 Stream + Kafka Bridge + InfluxDB". Although the patent CN110097159A describes a strategy that triggers LRU eviction and enables RocksDB disk overflow when Redis memory reaches 80%, all data still needs to be processed through a single Redis instance, which does not solve the bottleneck problem of single-threaded CPU processing. In a real-world test of 120,000 points / second, the system disk utilization of the edge node reached 100%, resulting in a 35% reduction in system lifespan.

[0025] The GE Predix Edge 2.2 uses TimescaleDB as its sole data storage endpoint. According to US20210123456A1 patent, this solution temporarily stores data in a local SQLite when write latency exceeds 500 milliseconds by collecting the SDK backpressure threshold, and then performs batch data recovery after the latency recovers. However, SQLite and TimescaleDB share the same disk, leading to intense disk contention. Furthermore, this solution relies solely on "write latency" for decision-making, failing to effectively eliminate dirty data caused by electromagnetic spikes, sensor disconnections, etc. This could cause the false positive rate for disconnections after PHM fine-tuning to increase from 2% to 9%.

[0026] Huawei's ROMA Edge 3.0 adds a "memory page pool" to the front end of the IoTDB. Patent CN114512728A proposes a strategy of writing data to disk in a 1-second time-slice aggregation order, combined with Netty's zero-copy approach. However, since the page pool and IoTDB reside in the same JVM, GC jitter can cause write latency to jump from 50 milliseconds to 800 milliseconds. Furthermore, the lack of an edge-level cache means that once the IoTDB goes offline, the acquisition thread will directly drop packets, potentially losing critical fault transition data within 30 seconds.

[0027] In summary, existing competing products have failed to overcome the architectural limitations of "single-level caching and single-point write," which provides significant room for innovation for the multi-level buffering and intelligent load balancing solution proposed in this patent. Our solution aims to effectively address the challenges encountered by existing solutions in data persistence by introducing multi-level caching and intelligent load balancing mechanisms, achieving zero packet loss, load balancing, and long-term traceability, thereby providing a more reliable and efficient solution for data management and analysis in the Industrial Internet of Things (IIoT).

[0028] To achieve the above objectives, see [link to relevant documentation]. Figure 1 , Figure 1 This application provides a flowchart illustrating an IoT data storage optimization method based on multi-level caching and intelligent traffic splitting, as shown in the following embodiment. Figure 1 As shown, the IoT data storage optimization method includes: S110: Construct a multi-level caching architecture between the data acquisition end and the persistent storage end of IoT data. The multi-level caching architecture includes at least local memory cache, distributed cache and database cache. S120: Based on the data characteristics and / or business needs of IoT data, the intelligent routing engine dynamically routes IoT data to the corresponding cache level in the multi-level cache architecture; S130: Perform data aggregation processing on IoT data and execute a timed batch write strategy to write the data to persistent storage. S140: Employs a data sharding and read / write separation strategy to distribute data across multiple database instances or nodes on the persistent storage side to achieve load balancing.

[0029] The IoT data storage optimization method based on multi-level caching and intelligent traffic distribution provided in this application effectively solves the challenges encountered by existing solutions in the data persistence process by introducing multi-level caching and intelligent traffic distribution mechanisms. It achieves zero packet loss, load balancing and long-term traceability of data, making it possible to persist massive amounts of sensor data with "high fidelity, load reduction and long cycle", thereby providing a more reliable and efficient solution for data management and analysis of industrial IoT.

[0030] in, Figure 1 In the illustrated embodiment, the above-described IoT data storage optimization method constructs a multi-level caching architecture between the IoT data acquisition end and the persistent storage end. This multi-level caching architecture includes at least a local memory cache, a distributed cache, and a database cache, comprising: In a multi-level caching architecture, a local memory cache is used as the first-level cache to store frequently accessed short-term data. In a multi-level caching architecture, a distributed cache cluster is used as a second-level cache to distribute data across multiple nodes, providing high availability and load balancing for data caching. In the multi-level caching architecture, a database cache is used as the third-level cache, which includes a time-series database and a relational database for persistent storage of IoT data.

[0031] in, Figure 1 In the illustrated embodiment, the above-mentioned IoT data storage optimization method, based on the data characteristics and / or business needs of IoT data, dynamically routes IoT data to the corresponding cache level in the multi-level cache architecture through an intelligent traffic routing engine, including: The intelligent routing engine makes data routing decisions based on at least one of the following factors: business priority, waveform signal data or data change frequency, data storage cost and access latency requirements.

[0032] in, Figure 1 In the illustrated embodiment, the above-mentioned IoT data storage optimization method performs data aggregation processing on IoT data and executes a timed batch write strategy to write the data to persistent storage, including: Downsampling and compression processing is performed on IoT data in waveform signal form; Based on the data requirements of the project management system, perform in-memory aggregation calculations on the feature value data; Based on the dynamic write frequency, the aggregated feature values ​​are written to persistent storage in batches; the dynamic write frequency is dynamically adjusted according to business needs.

[0033] in, Figure 1 In the illustrated embodiment, the IoT data storage optimization method described above employs a data sharding and read-write separation strategy, distributing data across multiple database instances or nodes on a persistent storage end, including: Hash sharding based on device ID, time range, or data tag; By using parallel processing, the hash-sharded data is distributed and stored across multiple database instances to achieve parallel data processing and load balancing.

[0034] in, Figure 1 In the illustrated embodiment, the above-mentioned IoT data storage optimization method employs a data sharding and read-write separation strategy to distribute data across multiple database instances or nodes on a persistent storage end, including: A master-slave replication architecture is used to achieve read-write separation of IoT data; the master-slave replication architecture includes a master database and a slave database, with the master database handling write operations and the slave database handling read operations. Combine caching systems to cache frequently accessed data in IoT data to improve read operation performance.

[0035] in, Figure 1 In the illustrated embodiment, the above-mentioned IoT data storage optimization method employs a data sharding and read-write separation strategy to distribute data across multiple database instances or nodes on a persistent storage end, including: IoT data is compressed and distributed across multiple nodes through distributed caching in a multi-level caching architecture; The compressed IoT data is written to a time-series database or relational database in a multi-level caching architecture through a locking mechanism.

[0036] in, Figure 1In the illustrated embodiment, the above-described method for optimizing IoT data storage employs a locking mechanism to prevent read / write conflicts when writing IoT data, including: A write lock is acquired before IoT data is written using a pre-defined configuration command in the database. The write lock is used to ensure that only one write operation occurs at a time. During reading, check the write lock status. If the write lock is held, wait or return an error message to avoid reading inconsistent data. Once the write operation is complete, release the write lock to allow read operations.

[0037] In summary, the technical solution provided by the embodiments of this application has the following advantages: (1) Multi-level caching mechanism adapts to different access needs and system load by storing and processing data at different levels.

[0038] Memory caching forms the crucial first level in a multi-level caching mechanism. It leverages the high-speed access capabilities of memory to provide an efficient storage path for data that requires frequent and rapid access. This caching layer primarily stores data that needs to be accessed quickly in the short term. This design not only optimizes the access speed of high-frequency data but also ensures the stability and reliability of the system.

[0039] In multi-node environments, distributed caching (Redis Cluster) is used as a secondary cache to improve the reliability and scalability of data storage. Distributed caching systems achieve high availability and load balancing by distributing data across multiple nodes. As a secondary cache, Redis not only provides fast data access but also enhances the system's fault tolerance through its built-in replication and sharding mechanisms.

[0040] In this architecture, the database acts as a three-tier cache, primarily used for persistent data storage. This includes relational databases (such as MySQL and PostgreSQL) and time-series databases (such as InfluxDB and TimescaleDB), which are responsible for long-term storage of processed, collected data to support complex queries and analysis. As a three-tier cache, the database ensures data persistence and consistency, preventing data loss even in the event of system restarts or node failures.

[0041] In summary, multi-level caching mechanisms, through the collaborative work of memory caching, distributed caching, and disk caching, provide flexible and efficient solutions for data storage and access in different scenarios.

[0042] (2) Data aggregation and batch storage are effective strategies to optimize data storage process and reduce database burden.

[0043] Data aggregation refers to processing raw data before writing it into a database to reduce data volume and improve data usability. For example... Figure 2 As shown, in a Project Management System (PHM), data aggregation is a key strategy for optimizing the efficiency of data storage and subsequent analysis, because PHM analysis typically focuses on long-term trends and pattern recognition rather than precise, instantaneous data points. For example, data can be aggregated every 10 minutes, hourly, or daily to balance the frequency and timeliness of data processing.

[0044] Scheduled batch writes reduce the frequency of database access by setting up scheduled tasks, such as performing data write operations every 10 minutes. This strategy significantly reduces the database load. To implement scheduled batch writes, Spring Scheduled Tasks is used to manage these tasks. To improve batch write performance, measures are taken such as using the database's batch insert API, adjusting the batch size, and sorting and deduplicating data before writing.

[0045] By implementing data aggregation and timed batch write strategies, the efficiency of data storage is significantly improved, the database load is reduced, and the stability and scalability of the entire system are enhanced.

[0046] (3) Load balancing can improve the performance and reliability of database systems when handling a large number of concurrent requests.

[0047] Data sharding is an effective load balancing technique that distributes data across multiple database instances or nodes. This approach significantly improves system processing capacity because it allows for parallel data processing, thereby reducing the load on individual databases.

[0048] Read / write splitting is another strategy to improve database concurrency. It separates read and write operations using a master-slave replication architecture. The master database handles all write operations, while the slave databases handle read operations. This architecture can improve database throughput because read operations are typically more frequent than write operations.

[0049] In summary, by carefully designing and implementing these load balancing strategies, it is possible to ensure that the database system remains efficient and stable when faced with a large number of concurrent requests, thereby supporting the continuous growth and changes in business.

[0050] As a second preferred embodiment, taking waveform signal data storage as an example, the storage optimization scheme of this application is as follows: Waveform signal data typically features high frequency and high data volume. For example, in industrial automation, sensors acquire waveform signal data at high frequencies. The following strategies are used for storing this type of data: Data downsampling and compression: Data downsampling techniques can reduce the number of data points while retaining critical information. For example, in industrial automation, for some non-critical waveform signal data, the sampling frequency can be reduced from 44K times per second to 4K times per second. This downsampling method usually does not significantly affect the detection results because mechanical faults almost always occur in the low-frequency signal range. By reducing the sampling frequency, the amount of data can be significantly reduced, thereby reducing storage costs and database load. Furthermore, using data compression algorithms, such as Huffman coding and LZ77, can further reduce data storage space. These algorithms can effectively reduce data redundancy and improve storage efficiency.

[0051] Local caching and batch writes: To reduce the pressure on the time-series database, the collected waveform signal data can be stored in a dedicated local cache, such as using Redis shards specifically for writing to the database. Redis's high performance and low latency make it an ideal caching choice. Data in the cache can then be written to the time-series database in batches via scheduled tasks or triggering mechanisms. For example, waveform signal data from a 2-second local cache can be written to a time-series database like InfluxDB every hour. This reduces the number of write operations to the database and improves database performance.

[0052] To prevent read-write conflicts, locks can be set when writing data. The specific implementation can follow these steps: Write lock: Before writing data, acquire a write lock, which can be achieved using the Redis SETNX command, ensuring that only one write operation can occur at a time; Read lock: When reading data, check the status of the write lock. If the write lock is held, wait or return an error message to avoid reading inconsistent data; Lock release: After the write operation is complete, release the write lock to allow other read operations.

[0053] Distributed caching and data sharding: When the business scale is large and a single database instance cannot meet the storage and processing requirements, a distributed caching system, such as ShardedRedis, can be used to distribute data across multiple cache nodes. Simultaneously, data sharding technology can be used to shard waveform signal data according to dimensions such as time range and device ID, storing it across multiple database instances. For example, hash sharding based on device ID can store waveform signal data from different devices on different database instances, thereby improving the system's scalability and processing capabilities.

[0054] As a third preferred embodiment, taking feature value data storage as an example, the data storage optimization is explained as follows: Eigenvalue data is typically obtained by processing and analyzing raw data. It is characterized by relatively small data volume but high requirements for real-time performance and accuracy. For example, in fields such as data analysis and machine learning, eigenvalue data is used for model training and prediction. The following strategies can be adopted for storing this type of data: In-memory aggregation calculations: These perform aggregation calculations on feature value data in memory, such as calculating the average, maximum, and minimum values. This reduces the amount of data that needs to be stored and improves data availability. For example, in a real-time data analysis system, the large amount of feature value data generated per second can be aggregated in memory, with the aggregation result calculated every 10 seconds.

[0055] Scheduled batch writes: Feature value data calculated in memory is written to the database in batches via a scheduled task. This reduces the frequency of database writes and lowers the database load. For example, the feature value aggregation results in memory can be written to a relational database such as MySQL every 5 minutes.

[0056] Read / write separation and caching strategy optimization: A master-slave replication architecture is adopted to achieve read / write separation, with the master database handling write operations and the slave database handling read operations. Simultaneously, caching strategies are used to optimize read operation performance, such as using caching systems like Memcached to cache frequently accessed data. For example, during machine learning model training, training data is stored in the master database, and model training is performed by reading data from the database. Meanwhile, frequently accessed feature values ​​are cached in Memcached to improve data retrieval speed.

[0057] In addition, such as Figure 3 As shown, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the IoT data storage optimization method based on multi-level caching and intelligent traffic splitting provided in any of the above embodiments.

[0058] like Figure 3As shown, the electronic device can include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory ROM 1002 or a program loaded from a storage device 1003 into a random access memory RAM 1004. The RAM 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 can operate the electronic device to exchange data with other devices via wireless or wired communication. Although the diagram shows a model building device with various systems, it should be understood that it is not required to implement or have all of the systems shown. It is possible to implement or have more or fewer systems alternatively.

[0059] In addition, this application also provides a computer storage medium storing a computer program thereon, which, when executed, implements the IoT data storage optimization method based on multi-level caching and intelligent traffic distribution provided by any of the above technical solutions.

[0060] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0061] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0062] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An IoT data storage optimization method based on multi-level caching and intelligent traffic distribution, characterized in that, include: A multi-level caching architecture is constructed between the data acquisition end and the persistent storage end of IoT data. The multi-level caching architecture includes at least local memory cache, distributed cache and database cache. Based on the data characteristics and / or business needs of the IoT data, the intelligent routing engine dynamically routes the IoT data to the corresponding cache level in the multi-level cache architecture. The IoT data is aggregated and processed, and a timed batch write strategy is executed to write it to persistent storage. By employing a data sharding and read-write separation strategy, data is distributed and stored across multiple database instances or nodes on the persistent storage end to achieve load balancing.

2. The method according to claim 1, characterized in that, The multi-level caching architecture constructed between the data acquisition end and the persistent storage end of IoT data includes at least local memory cache, distributed cache, and database cache, including: In the multi-level caching architecture, a local memory cache is used as the first-level cache to store frequently accessed short-term data. In the multi-level caching architecture, a distributed cache cluster is used as a second-level cache to distribute data among multiple nodes, providing high availability and load balancing for data caching. In the multi-level caching architecture, a database cache is used as a third-level cache, which includes a time-series database and a relational database for persistent storage of the IoT data.

3. The method according to claim 1, characterized in that, The step of dynamically routing the IoT data to the corresponding cache layer in the multi-level caching architecture through an intelligent traffic routing engine, based on the data characteristics and / or business needs of the IoT data, includes: The intelligent routing engine makes data routing decisions based on at least one of the following factors: business priority, waveform signal data or data change frequency, data storage cost, and access latency requirements.

4. The method according to claim 1, characterized in that, The step of performing data aggregation processing on the IoT data and executing a timed batch write strategy to write it to persistent storage includes: The IoT data in waveform signal form is downsampled and compressed; Based on the data requirements of the project management system, perform in-memory aggregation calculations on the feature value data; According to the dynamic write frequency, the aggregated feature values ​​are written to the persistent storage terminal in batch write mode; wherein, the dynamic write frequency is dynamically adjusted according to business needs.

5. The method according to claim 1, characterized in that, The data sharding and read / write separation strategy employed distributes data across multiple database instances or nodes on the persistent storage end, including: Hash sharding based on device ID, time range, or data tag; By using parallel processing, the hash-sharded data is distributed and stored across multiple database instances to achieve parallel data processing and load balancing.

6. The method according to claim 1, characterized in that, The method employs a data sharding and read-write separation strategy to distribute data across multiple database instances or nodes on the persistent storage end, including: The read-write separation of the IoT data is achieved using a master-slave replication architecture; wherein the master-slave replication architecture includes a master database and a slave database, the master database handles write operations, and the slave database handles read operations; By combining a caching system with the caching of frequently accessed data in the IoT data, read operation performance can be improved.

7. The method according to claim 1, characterized in that, The method employs a data sharding and read-write separation strategy to distribute data across multiple database instances or nodes on the persistent storage end, including: The IoT data is compressed and distributed across multiple nodes through the distributed cache in the multi-level caching architecture. The compressed IoT data is written to the time-series database or relational database in the multi-level caching architecture through a locking mechanism.

8. The method according to claim 1 or 7, characterized in that, A locking mechanism is used to prevent read / write conflicts when writing the IoT data, including: A write lock is acquired before the IoT data is written using a pre-defined setting command in the database. The write lock is used to ensure that only one write operation occurs at a time. During reading, the write lock status is checked. If the write lock is occupied, the system waits or returns an error message to avoid reading inconsistent data. Once the write operation is complete, the write lock is released, allowing read operations to proceed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the IoT data storage optimization method based on multi-level caching and intelligent traffic splitting as described in any one of claims 1-8.

10. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the IoT data storage optimization method based on multi-level caching and intelligent traffic splitting as described in any one of claims 1 to 8.

Images