A multi-level caching method, system, medium, and computer device for massive measurement points

Abstract

This invention relates to a multi-level caching method, system, medium, and computer device for massive measurement points in the field of massive measurement point technology. The method includes acquiring raw data packets reported by measurement points at a preset frequency; routing the measurement points and real-time sampled value data obtained from parsing the raw data packets to the corresponding target nodes of the first-level cache according to preset business grouping rules and a parallel writing mechanism, and writing them into a dedicated storage structure within the target nodes; periodically exporting the real-time sampled value data in the dedicated storage structure to a disk-based second-level persistent cache in the form of data blocks; and, based on the measurement point identifier in the data query request, first reading the target sampled value data from the target node in the first-level cache; if it does not exist, retrieving it from the second-level persistent cache. This invention logically segments and distributes the data traffic of massive measurement points to different cache nodes, achieving horizontal scaling of processing capacity and improving the system's operational efficiency and overall stability.

Description

Technical Field

[0001] This invention relates to a multi-level caching method, system, medium, and computer device for massive measurement points, belonging to the field of massive measurement point technology. Background Technology

[0002] With the rapid development of the Internet of Things, smart grids, and industrial automation control, the number of physical monitoring points connected to monitoring systems has exploded. From traditional substation equipment monitoring to large-scale urban infrastructure management, the scale of monitoring points has increased from tens of thousands to millions or even hundreds of millions, and the data reporting frequency has increased from minutes to seconds or even milliseconds. This data characteristic of "massive monitoring points and high-frequency sampling" places extremely stringent requirements on the write throughput, storage efficiency, and query response speed of the backend data processing system.

[0003] Traditional solutions for processing measurement point data often employ relational databases or single real-time databases. When processing massive amounts of time-series data, these solutions generally suffer from the following technical bottlenecks: First, the write bottleneck is a prominent issue. High-frequency concurrent writes to a massive number of measurement points can easily lead to disk I / O saturation. Traditional architectures with a single write node or simple table partitioning strategies are unable to support millions of data write requests per second, resulting in data backlog and reduced real-time performance.

[0004] Secondly, query performance deteriorates sharply with increasing data volume. Due to the lack of a multi-level indexing mechanism tailored to the characteristics of measurement point data, traditional B-tree indexes suffer from severe index bloat and low query efficiency when dealing with "time range queries" involving tens of millions of measurement points and long time spans. When historical data needs to be queried, frequent disk scans result in response delays of several seconds or even minutes, failing to meet the needs of real-time monitoring and rapid fault location.

[0005] Furthermore, the system suffers from poor scalability and significant resource waste. Existing solutions typically lack granular data routing rules and elastic scaling capabilities. When the number of measurement points increases or business groups are adjusted, large-scale data redistribution of the entire storage cluster is often required, sometimes even necessitating system downtime for expansion. Simultaneously, due to the lack of a targeted real-time sample value structure and a hot / cold data tiering mechanism, a large amount of "hot data" and "cold data" are stored together, consuming valuable memory resources and failing to guarantee query efficiency for frequently accessed data. Cache hit rates drop sharply as the system scales up.

[0006] Therefore, how to achieve low-latency data writing, high-hit-rate caching, and efficient time-range querying in scenarios with massive data points, while supporting elastic and smooth cluster expansion, has become a critical issue that urgently needs to be addressed in the field of data processing technology. No solutions have yet been proposed to address these technical challenges. Summary of the Invention

[0007] The purpose of this invention is to propose a multi-level caching method, system, medium, and computer device for massive measurement points. This method logically segments and distributes the data traffic of massive measurement points to different cache nodes, avoiding the performance bottleneck of a single node, realizing horizontal expansion of processing capabilities, transforming string search into numerical calculation through multi-level indexing, significantly reducing the search range, improving space utilization, minimizing memory and disk storage costs, and improving the system's operational efficiency and overall stability.

[0008] To achieve the above objectives, the present invention is implemented using the following technical solution: Firstly, this invention proposes a multi-level caching method for massive measurement points, including: Acquire the raw data packets reported by the measuring points at a preset frequency, and parse the raw data packets to extract the measuring point identifier, real-time sampled value data and timestamp; According to the preset business grouping rules, the measurement point identifier and real-time sampled value data are routed to the target node in the target node cluster corresponding to the first-level cache; The real-time sampled data is written to the dedicated storage structure inside the target node through a parallel writing mechanism. The real-time sampled value data in the target node of the first-level cache is periodically exported to the second-level persistent cache based on disk in the form of data blocks; The dedicated storage structure is built based on a memory allocation and management mechanism and includes multi-level indexes. These multi-level indexes include a first-level index, a second-level index, and a third-level index. The first-level index establishes a mapping relationship between measurement point identifiers and internal numerical numbers, using a segmented mapping table to map the measurement point identifiers to internal numerical numbers. The second-level index is used to construct a hash index or B+ tree index based on the internal numerical numbers to locate the corresponding memory storage block for real-time sampled value data. The third-level index is used to construct an ordered index based on timestamps or sampling sequence numbers based on the memory storage blocks to retrieve real-time sampled value data at a target time point.

[0009] Furthermore, the business grouping rules are based on at least one predefined factor among the physical affiliation of the measurement point, data type, and monitoring area.

[0010] Further, the step of routing the measurement point identifier and real-time sampled value data to the target node in the target node cluster corresponding to the first-level cache according to the preset business grouping rules includes: The dynamic routing table is constructed according to the preset business grouping rules. The dynamic routing table records the mapping relationship between business groups and target nodes of the first-level cache. The service group attribute to which the measurement point belongs is determined based on the metadata of the measurement point and the service grouping rules; the metadata includes at least one of physical location, data type, and monitoring area; Based on the service group attributes of the test point, query the dynamic routing table to determine the target node responsible for processing the test point; When the target node cluster of the first-level cache is expanded or reduced, the mapping relationship of the affected service groups in the dynamic routing table is updated.

[0011] Furthermore, the step of writing the real-time sampled value data into the dedicated storage structure inside the target node through a parallel writing mechanism includes: Several independent memory blocks are pre-allocated as write channels, each write channel corresponding to a set of measurement point identifiers or a data stream, wherein each write channel internally employs a lock-free data structure or atomic operations; Based on the measurement point identifier of the real-time sampled data, the real-time sampled data is allocated to the write channel corresponding to the dedicated storage structure inside the target node.

[0012] Furthermore, before writing to the dedicated storage structure, the method further includes optimizing the storage of the real-time sampled value data, including: Based on the type and accuracy requirements of the real-time sampled data corresponding to the measuring point, an encoding algorithm is used for compression and storage. The encoding algorithm includes one or more combinations of difference encoding, travel length encoding, and fixed-point to floating-point encoding. The compressed data is then subjected to a second compression using a general compression algorithm, and the second-compressed data blocks are stored in the dedicated storage structure.

[0013] Furthermore, the second-level persistent cache maintains a data index corresponding to the memory of the target node of the first-level cache, which is used to handle historical data requests whose query time range exceeds the memory capacity of the first-level cache.

[0014] Secondly, this invention proposes a multi-level caching system for massive measurement points, comprising: The data access module is used to acquire the raw data packets reported by the measuring points at a preset frequency, and to parse the raw data packets to extract the measuring point identifier, real-time sampled value data and timestamp. The data routing module routes the measurement point identifier and real-time sampled value data to the target node in the target node cluster corresponding to the first-level cache according to the preset business grouping rules. A parallel write module is used to write the real-time sampled value data into a dedicated storage structure inside the target node via a parallel write mechanism. This dedicated storage structure is built based on a memory allocation and management mechanism and includes multi-level indexes. The multi-level indexes include a first-level index, a second-level index, and a third-level index. The first-level index is used to establish a mapping relationship between measurement point identifiers and internal numerical numbers, using a segmented mapping table to map the measurement point identifiers to internal numerical numbers. The second-level index is used to construct a hash index or a B+ tree index based on the internal numerical numbers to locate the corresponding memory storage block of the real-time sampled value data. The third-level index is used to construct an ordered index based on timestamps or sampling sequence numbers based on the memory storage blocks to retrieve the real-time sampled value data at the target time point. The data persistence module is used to periodically export the real-time sampled value data in the target node of the first-level cache to the disk-based second-level persistent cache in the form of data blocks.

[0015] Furthermore, the system also includes a data query engine, which responds to data query requests and, based on the measurement point identifier and time range in the request, sequentially locates and reads the target sampled value data from the target node corresponding to the first-level cache unit through the multi-level index.

[0016] Thirdly, the present invention proposes a computer-readable storage medium storing a computer program / instruction thereon, characterized in that, when the computer program / instruction is executed by a processor, it implements the steps of the multi-level caching method for massive measurement points.

[0017] Fourthly, the present invention provides a computer device comprising: Memory, used to store computer programs / instructions; A processor is used to execute the computer program / instructions to implement the multi-level caching method for massive measurement points.

[0018] Compared with the prior art, the beneficial effects of the present invention are: This invention uses preset business grouping rules to logically segment and distribute the data traffic of massive measurement points to different cache nodes, avoiding the performance bottleneck of a single node and realizing horizontal scaling of processing capabilities. Inside the node, a dedicated storage structure based on memory pre-allocation and parallel write channels are used to eliminate the overhead caused by traditional lock contention, and can easily cope with high-frequency, large-scale real-time data write surges. This invention transforms string searching into numerical calculation by setting up multi-level indexes, which greatly reduces the search range and enables nanosecond-level fast positioning in massive data, meeting the dual query requirements of real-time monitoring and historical data backtracking; hot data is retrieved first from memory, while cold data is automatically degraded to disk, and the query engine can transparently access across levels, which not only ensures millisecond-level response for hot data, but also provides access to the full historical data. This invention employs an array- and paging-based memory allocation and management mechanism to provide storage for measurement point data, reducing memory fragmentation and improving space utilization. Before writing data, it first performs preliminary compression using targeted algorithms such as differential encoding based on data type, and then performs secondary compression using general compression algorithms. This can minimize memory and disk storage costs while ensuring query performance, and significantly extend the residence time of hot data in memory. This invention uses a service group mapping mechanism based on a dynamic routing table, which allows the cluster to expand or shrink by migrating or updating the mapping relationship of only a few affected groups, without rehashing all data or large-scale data migration, greatly improving the system's operation and maintenance efficiency. The measurement point data of different service groups are isolated in different nodes or processing channels, so the failure of a single node or abnormal traffic of a certain service group will not spread to the entire cluster, thus improving the overall stability of the system. This invention reduces memory and disk usage through compression algorithms; through multi-level caching, it reduces reliance on expensive memory resources, allowing more data to be stored on inexpensive disk media, ensuring that resources are dynamically adjusted according to business volume and avoiding resource waste. Attached Figure Description

[0019] Figure 1 This is a flowchart of a multi-level caching method for massive measurement points proposed in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use.

[0021] Example 1:

[0022] This embodiment proposes a multi-level caching method for massive measurement points, such as... Figure 1 As shown, it includes: The system acquires raw data packets reported by measurement points at a preset frequency, parses the raw data packets, and extracts the measurement point identifier, real-time sampled value data, and timestamp. Among them, massive measurement points refer to a large number of sensors, smart meters, or devices, and each measurement point has a unique ID. The preset frequency can be once per second (1Hz) or higher. Assuming there are 10 million measurement points, and each reports once per second, then 10 million writes need to be processed per second (10M TPS). According to the preset business grouping rules, the measurement point identifier and the corresponding real-time sampled value data are routed to the target node in the target node cluster corresponding to the first-level cache; wherein, the business grouping rules are based on at least one of the following predefined criteria: physical affiliation of the measurement point, data type, and monitoring area; specifically, the business grouping routing is based on physical affiliation (e.g., substation A, factory B) and data type (e.g., temperature, voltage). The parsed real-time sampled data is written to a dedicated storage structure inside the target node through a parallel writing mechanism. The real-time sampled value data in the target node of the first-level cache is periodically exported to the disk-based second-level persistent cache in the form of data blocks; The dedicated storage structure is built on a memory allocation and management mechanism and includes multi-level indexes to enable fast location and retrieval of measurement point data. The first-level index is used to establish the mapping relationship between the measurement point identifier and the internal number. It adopts the form of a segmented mapping table to map the measurement point identifier to the internal number. The second-level index is used to build a hash index or B+ tree index based on the internal numbering to locate the memory storage block of the corresponding real-time sampled value data. The third-level index is used to construct an ordered index based on timestamps or sampling sequence numbers based on the memory storage block, and to retrieve real-time sampled value data at the target time point.

[0023] In this embodiment, in response to a data query request, based on the measurement point identifier in the request, the corresponding numerical number is first obtained through the first-level index. Then, based on the numerical number and time range, the target sampled value data is located and read from the corresponding cache node through the second-level index and the third-level index in sequence. If the required data is not found in the first-level cache, the retrieval continues from the second-level persistent cache.

[0024] In this embodiment, according to preset business grouping rules, the measurement points and the real-time sampled value data obtained by parsing are routed to the target node in the corresponding first-level cache node cluster, including: Maintain a dynamic routing table that records the mapping relationship between service groups and first-level cache nodes; The service group attribute to which the measurement point belongs is determined based on the metadata of the measurement point and the service grouping rules; the metadata includes at least one of physical location, data type, and monitoring area; Based on the service group attributes of the test point, query the dynamic routing table to determine the target node responsible for processing the test point. When the target node cluster of the first-level cache is expanded or reduced, only the mapping relationship of the affected service groups in the dynamic routing table needs to be updated, so as to achieve elastic scaling of the cluster without the business's awareness or with low awareness.

[0025] In this embodiment, the parsed real-time sampled value data is written to a dedicated storage structure inside the target node through a parallel writing mechanism, including: Several independent memory blocks are pre-allocated as write channels, and each write channel corresponds to a set of measurement point identifiers or a data stream; The parsed real-time sampled data is allocated to the corresponding write channel according to its measurement point identifier; Within each write channel, lock-free data structures or atomic operations are used to achieve lock-free concurrent data writing. In this embodiment, lock-free concurrent writing is achieved by distributing data to different independent channels according to measurement points. Each channel has its own dedicated memory block. If concurrency is still required within a channel, a compare-and-swap instruction or a lock-free queue is used to implement writing, completely avoiding thread blocking.

[0026] In this embodiment, before writing to the dedicated storage structure, the method further includes optimizing the storage of real-time sampled value data, including: Based on the real-time sampled data corresponding to the measurement point, memory space is compressed. According to the type and accuracy requirements of the real-time sampled data, a targeted encoding algorithm is used for compression and storage. The encoding algorithm includes one or more combinations of difference encoding, travel length encoding, and fixed-point to floating-point encoding. The compressed data is then subjected to a second compression using a general compression algorithm, and the second-compressed data blocks are stored in a dedicated storage structure.

[0027] In this embodiment, the second-level persistent cache maintains a data index corresponding to the memory of the target node in the first-level cache, used to handle historical data requests whose query time range exceeds the memory capacity of the first-level cache. When the data block in memory is full, it will be asynchronously flushed to disk; during a query, if the data cannot be found in the memory of the first-level cache, it will be searched in the solid-state drive of the second-level persistent cache according to the index, which ensures both high-speed access to hot data and the traceability of historical data.

[0028] Example 2:

[0029] This embodiment proposes a multi-level caching system for massive measurement points based on Embodiment 1, including: The data access module is used to acquire the raw data packets reported by the measuring points at a preset frequency, and to parse the raw data packets to extract the measuring point identifier, real-time sampled value data and timestamp. The data routing module routes the measurement point identifier and real-time sampled value data to the target node in the target node cluster corresponding to the first-level cache according to the preset business grouping rules. A parallel write module is used to write the real-time sampled value data into a dedicated storage structure inside the target node via a parallel write mechanism. This dedicated storage structure is built based on a memory allocation and management mechanism and includes multi-level indexes. The multi-level indexes include a first-level index, a second-level index, and a third-level index. The first-level index is used to establish a mapping relationship between measurement point identifiers and internal numerical numbers, using a segmented mapping table to map the measurement point identifiers to internal numerical numbers. The second-level index is used to construct a hash index or a B+ tree index based on the internal numerical numbers to locate the corresponding memory storage block of the real-time sampled value data. The third-level index is used to construct an ordered index based on timestamps or sampling sequence numbers based on the memory storage blocks to retrieve the real-time sampled value data at the target time point. The data persistence module is used to periodically export the real-time sampled value data in the target node of the first-level cache to the disk-based second-level persistent cache in the form of data blocks.

[0030] In this embodiment, the system further includes a data query engine, which is used to respond to data query requests and, based on the measurement point identifier and time range in the request, sequentially locate and read the target sampled value data from the target node corresponding to the first-level cache unit through the multi-level index.

[0031] Example 3:

[0032] This embodiment proposes a computer-readable storage medium based on Embodiment 1, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, it implements the steps of the multi-level caching method for massive measurement points.

[0033] Example 4:

[0034] This embodiment proposes a computer device based on Embodiment 1, comprising: Memory, used to store computer programs / instructions; A processor is used to execute computer programs / instructions to implement the steps of the described multi-level caching method for massive measurement points.

[0035] Example 5:

[0036] Based on Example 1, this embodiment further describes the specific application of the multi-level caching method for massive measurement points in the real-time monitoring scenario of power systems, in order to verify the effectiveness of the method.

[0037] A large-scale smart city power grid monitoring system needs to monitor 5 million smart meters deployed throughout the city in real time, achieving real-time monitoring of a massive number of measuring points. Each meter reports real-time sampled data such as voltage, current, and power at a frequency of once per minute (preset frequency). The system needs to support real-time monitoring display on a large screen, and also be able to retrieve historical data from the past week or even a month for load curve analysis.

[0038] Step 1: Data Access and Dynamic Routing At 00:00:01, the system received the real-time current value reported by meter "A-101" (measuring point identifier).

[0039] Based on the meter's physical location (substation area, such as "Chengbei Substation") and data type (such as "current data"), the system finds the predefined business group as "Chengbei-Current".

[0040] The data access layer queries the dynamic routing table and finds that the target node of the first-level cache corresponding to the "Chengbei-Dianliu" service group is the node "Cache-Node-3". This data is quickly routed to Cache-Node-3 for processing.

[0041] Step 2: Parallel writes and multi-level index construction within the target node of the first-level cache The data arrives at the "Cache-Node-3" node. The node pre-allocates 16 independent memory blocks as write channels. Based on the hash value of the test point ID "A-101", the system assigns it to channel number 5. Channel 5 uses a lock-free data structure to directly append data, avoiding performance overhead caused by multi-threaded contention for lock resources. Simultaneously, the node allocates storage space for the latest data in its dedicated memory block for test point "A-101".

[0042] The system quickly converts the lengthy measurement point ID "A-101" into a shorter, easier-to-process internal number (e.g., 10001) through a segmented mapping table, and uses this internal number as the first-level index.

[0043] The system uses the number 10001 as the key, finds the starting memory block address where all data for this measurement point is stored in memory through a hash index, and uses the starting memory block address as the second-level index.

[0044] Within this memory block, the data is indexed in an ordered manner according to the timestamp (e.g., "2026-03-11 00:00:01"), and this ordered index is used as a third-level index to ensure that the sampled value at any point in time can be quickly located.

[0045] Step 3: Data Query Response Scenario A: Real-time monitoring dashboard query The monitoring screen requests a query for the current real-time current value of meter "A-101".

[0046] The data query engine received the request and, based on the measurement point ID "A-101" and the time range "last minute", sequentially used the first index, second index, and third-level index to directly locate and return the latest current data in the memory of "Cache-Node-3" in milliseconds, thus ending the query.

[0047] Scenario B: Historical Data Analysis and Query Engineers need to analyze the load curve data of meter "A-101" over the past week.

[0048] The query engine first searches in the dedicated storage structure of the first-level cache according to the steps of scenario A, but only the hot data of the most recent 2 hours is stored in memory, and the earlier data does not exist.

[0049] The system determines that "the required data does not exist in the first-level cache" and automatically redirects to the second-level persistent cache stored on disk. The system also utilizes well-maintained data indexes to quickly retrieve historical data blocks from the disk for the past week and returns them to engineers for analysis.

[0050] Core mechanism operation effect: Due to the addition of a new "Chengdong Substation", the system needed to be expanded. The maintenance personnel added a new node "Cache-Node-5" and updated the dynamic routing table, pointing the "Chengdong-Voltage" and "Chengdong-Current" business groups to the new node. During the entire expansion process, the data routing of other business groups (such as "Chengbei-Current") was not affected in any way, realizing cluster scaling without the business's awareness.

[0051] The current value reported by meter "A-101" fluctuates relatively little. Before writing to the storage structure, the system first performs differential encoding on several consecutive sampling points (e.g., 10.1A, 10.1A, 10.2A, 10.1A) (only recording the first value and subsequent differences), and then performs secondary compression on the encoded byte stream using the LZ4 general compression algorithm. Ultimately, the data that originally required 32 bytes is compressed to only 8 bytes and stored in memory, greatly improving memory utilization.

[0052] In this embodiment, the data access and routing module is responsible for receiving, parsing, and routing in step 1.

[0053] The multi-level caching module includes a first-level cache unit built on a memory allocation and management mechanism, and a second-level persistent cache unit built on persistent storage. The first-level cache node cluster of the first-level cache unit contains multiple nodes such as "Cache-Node-3", which are responsible for performing the writing and index building in step 2; the second-level persistent cache unit is responsible for storing historical data and supporting historical data querying in step 3.

[0054] Data Query Engine: Responsible for processing query requests in step 3 and coordinating the retrieval of data from various levels of cache.

[0055] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0056] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0057] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0058] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0059] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A multi-level caching method for massive measurement points, characterized in that, include: Acquire the raw data packets reported by the measuring points at a preset frequency, and parse the raw data packets to extract the measuring point identifier, real-time sampled value data and timestamp; According to the preset business grouping rules, the measurement point identifier and real-time sampled value data are routed to the target node in the target node cluster corresponding to the first-level cache; The real-time sampled data is written to the dedicated storage structure inside the target node through a parallel writing mechanism. The real-time sampled value data in the target node of the first-level cache is periodically exported to the second-level persistent cache based on disk in the form of data blocks; The dedicated storage structure is built based on a memory allocation and management mechanism and includes multi-level indexes. These multi-level indexes include a first-level index, a second-level index, and a third-level index. The first-level index establishes a mapping relationship between measurement point identifiers and internal numerical numbers, using a segmented mapping table to map the measurement point identifiers to internal numerical numbers. The second-level index is used to construct a hash index or B+ tree index based on the internal numerical numbers to locate the corresponding memory storage block for real-time sampled value data. The third-level index is used to construct an ordered index based on timestamps or sampling sequence numbers based on the memory storage blocks to retrieve real-time sampled value data at a target time point.

2. The multi-level caching method for massive measurement points according to claim 1, characterized in that, The business grouping rules are based on at least one of the following predefined criteria: physical affiliation of the measurement point, data type, and monitoring area.

3. The multi-level caching method for massive measurement points according to claim 2, characterized in that, The step of routing the measurement point identifier and real-time sampled value data to the target node in the target node cluster corresponding to the first-level cache according to the preset business grouping rules includes: The dynamic routing table is constructed according to the preset business grouping rules. The dynamic routing table records the mapping relationship between business groups and target nodes of the first-level cache. The service group attribute to which the measurement point belongs is determined based on the metadata of the measurement point and the service grouping rules; the metadata includes at least one of physical location, data type, and monitoring area; Based on the service group attributes of the test point, query the dynamic routing table to determine the target node responsible for processing the test point; When the target node cluster of the first-level cache is expanded or reduced, the mapping relationship of the affected service groups in the dynamic routing table is updated.

4. The multi-level caching method for massive measurement points according to claim 1, characterized in that, The step of writing the real-time sampled value data into the dedicated storage structure inside the target node through a parallel writing mechanism includes: Several independent memory blocks are pre-allocated as write channels, each write channel corresponding to a set of measurement point identifiers or a data stream, wherein each write channel internally employs a lock-free data structure or atomic operations; Based on the measurement point identifier of the real-time sampled data, the real-time sampled data is allocated to the write channel corresponding to the dedicated storage structure inside the target node.

5. The multi-level caching method for massive measurement points according to claim 4, characterized in that, Before writing to the dedicated storage structure, the method further includes storage optimization of the real-time sampled value data, including: Based on the type and accuracy requirements of the real-time sampled data corresponding to the measuring point, an encoding algorithm is used for compression and storage. The encoding algorithm includes one or more combinations of difference encoding, travel length encoding, and fixed-point to floating-point encoding. The compressed data is then subjected to a second compression using a general compression algorithm, and the second-compressed data blocks are stored in the dedicated storage structure.

6. The multi-level caching method for massive measurement points according to claim 1, characterized in that, The second-level persistent cache maintains a data index corresponding to the memory of the target node of the first-level cache, which is used to handle historical data requests whose query time range exceeds the memory capacity of the first-level cache.

7. A multi-level caching system for massive measurement points, characterized in that, The system is configured to perform the method of any one of claims 1 to 6, comprising: The data access module is used to acquire the raw data packets reported by the measuring points at a preset frequency, and to parse the raw data packets to extract the measuring point identifier, real-time sampled value data and timestamp. The data routing module routes the measurement point identifier and real-time sampled value data to the target node in the target node cluster corresponding to the first-level cache according to the preset business grouping rules. A parallel write module is used to write the real-time sampled value data into a dedicated storage structure inside the target node via a parallel write mechanism. This dedicated storage structure is built based on a memory allocation and management mechanism and includes multi-level indexes. The multi-level indexes include a first-level index, a second-level index, and a third-level index. The first-level index is used to establish a mapping relationship between measurement point identifiers and internal numerical numbers, using a segmented mapping table to map the measurement point identifiers to internal numerical numbers. The second-level index is used to construct a hash index or a B+ tree index based on the internal numerical numbers to locate the corresponding memory storage block of the real-time sampled value data. The third-level index is used to construct an ordered index based on timestamps or sampling sequence numbers based on the memory storage blocks to retrieve the real-time sampled value data at the target time point. The data persistence module is used to periodically export the real-time sampled value data in the target node of the first-level cache to the disk-based second-level persistent cache in the form of data blocks.

8. The multi-level caching system for massive measurement points according to claim 7, characterized in that, The system also includes a data query engine, which responds to data query requests and, based on the measurement point identifier and time range in the request, sequentially locates and reads the target sampled value data from the target node of the corresponding first-level cache unit through the multi-level index.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the multi-level caching method for massive measurement points as described in any one of claims 1 to 6.

10. A computer device, characterized in that, include: Memory, used to store computer programs / instructions; A processor for executing the computer program / instructions to implement the steps of the multi-level caching method for massive measurement points as described in any one of claims 1 to 6.

Images