A time series database interaction system and method for power generation enterprise report

Abstract

The application provides a time series database interaction system and method for power generation enterprise report forms, the system of the application comprises: a partition management unit, configured to split time series data into a plurality of continuous and mutually non-overlapping data blocks; a state management unit, configured to maintain corresponding state description information for each data block; a cache storage unit, configured to cache data with data blocks as storage granularity, and associate each cache entry with the state description information of the data block corresponding to the cache data; a refresh control unit, configured to determine whether the cache data of any data block is available in response to access of the data block by a data reading operation, and trigger a data reloading operation for the data block if the cache data is not available; and a query processing unit, configured to receive and parse a data query instruction issued by a report form system, disassemble the data query instruction into a plurality of sub-query tasks, schedule execution of the sub-query tasks, and feed back the execution results of the sub-query tasks to the report form system after merging.

Description

Technical Field

[0001] This invention relates to the field of power system automation and data processing technology, and in particular to a time-series database interaction system and method for power generation enterprise reports. Background Technology

[0002] The production and operation of power generation companies (including thermal power plants, hydropower plants, wind farms, and photovoltaic power stations) are highly dependent on the massive amounts of time-series data stored in real-time databases. The management, statistics, and reporting of these companies rely on reporting systems, requiring the periodic extraction of time-series data from real-time databases for aggregation and statistical analysis to generate daily, monthly, quarterly, and annual production reports.

[0003] In the actual production environment of power generation enterprises, time-series data often experiences delays in arrival due to factors such as unstable communication networks, equipment restarts, and manual data entry, a phenomenon known as data backfilling. For example, after a communication interruption is restored, data from several hours ago may only be written to the real-time database. This results in data from the same time window being added at multiple different points in time.

[0004] Existing technologies have the following technical drawbacks when handling data backfilling: First, when data backfilling occurs, the caching system has difficulty detecting data changes, resulting in report queries returning expired data; Second, even if data changes are detected, existing solutions typically use a full refresh strategy, which requires re-acquiring and recalculating a large amount of data, resulting in huge overhead; Third, when multiple queries hit the same expired cache at the same time, each triggers a refresh independently, causing duplicate calculations and wasted resources. Summary of the Invention

[0005] The purpose of this invention is to provide a time-series database interaction system and method for power generation enterprise reports, aiming to solve the above-mentioned problems in the prior art.

[0006] This invention provides a time-series database interaction system for power generation enterprise reports, comprising: The partition management unit is used to divide time-series data into several continuous and non-overlapping data blocks according to a preset time granularity, and generate a unique identifier for each data block. A status management unit, connected to the partition management unit, is used to maintain corresponding status description information for each data block; wherein, the status description information includes a first type of information reflecting the degree of data change and a second type of information reflecting data availability; A cache storage unit, connected to the state management unit, is used to cache data at the data block level and associate the state description information of the data block corresponding to the cached data with each cache entry. A refresh control unit, connected to the state management unit and the cache storage unit, is used to respond to a data read operation accessing any of the data blocks, determine whether the cached data of the data block is available based on the second type of information of the data block, and if it is not available, trigger a data reload operation for the data block. The query processing unit, connected to the cache storage unit and the reporting system, is used to receive and parse the data query instructions issued by the reporting system, break down the data query instructions into several sub-query tasks according to the time range involved, each sub-query task points to a corresponding data block, schedule the execution of each sub-query task, and merge the execution results of each sub-query task and feed them back to the reporting system.

[0007] This invention provides a time-series database interaction method for power generation enterprise reports, including: The partition management unit divides the time series data into several continuous and non-overlapping data blocks according to the preset time granularity, and generates a unique identifier for each data block. The status management unit maintains corresponding status description information for each data block; wherein, the status description information includes a first type of information reflecting the degree of data change and a second type of information reflecting data availability; The cache storage unit caches data at the data block level, and records the status description information of the data block corresponding to the cached data for each cache entry. The query processing unit receives and parses the data query instructions issued by the reporting system, and breaks down the data query instructions into several sub-query tasks according to the time range involved. Each sub-query task points to the corresponding data block. By refreshing the control unit in response to the data reading operation of each sub-query task on the corresponding data block, the cached data of the data block is determined based on the second type of information of the data block. If it is not available, a data reload operation for the data block is triggered. The query processing unit schedules the execution of each sub-query task and merges the execution results of each sub-query task before feeding them back to the reporting system.

[0008] This invention also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the above-described time-series database interaction method for power generation enterprise reports.

[0009] This invention also provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor, implements the steps of the aforementioned time-series database interaction method for power generation enterprise reports.

[0010] The following beneficial effects can be achieved by adopting the embodiments of the present invention: The embodiments of the present invention propose a time-series data interaction system between a power generation enterprise reporting system and a real-time database, so as to effectively solve the problems of difficulty in maintaining the consistency of cached data and excessive overhead of full refresh in the data backfill scenario. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of a time-series database interaction system for power generation enterprise reports according to an embodiment of the present invention; Figure 2 This is a flowchart of a time-series database interaction method for power generation enterprise reports according to an embodiment of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0014] System Implementation Examples According to embodiments of the present invention, a time-series database interaction system for power generation enterprise reports is provided. Figure 1 This is a schematic diagram of a time-series database interaction system for power generation enterprise reports according to an embodiment of the present invention, such as... Figure 1 As shown, the time-series database interaction system for power generation enterprise reports according to an embodiment of the present invention specifically includes: Partition management unit 10 is used to divide time-series data into several continuous and non-overlapping data blocks according to a preset time granularity, and generate a unique identifier for each data block; the partition management unit specifically includes: The granularity configuration subunit is used to configure the currently used time granularity; A boundary calculation subunit, connected to the granularity configuration subunit, is used to obtain the time granularity and calculate the boundary information of the data block to which the time stamp to be processed belongs based on the timestamp to be processed and the time granularity. An index maintenance subunit, connected to the boundary calculation subunit, is used to obtain the boundary information and construct an ordered index using the starting boundary in the boundary information as the key; wherein, the ordered index is used to locate the corresponding data block according to any timestamp, and to locate all data blocks covering any time range according to any time range.

[0015] A status management unit 12, connected to the partition management unit, is used to maintain corresponding status description information for each data block; wherein, the status description information includes a first type of information reflecting the degree of data change and a second type of information reflecting data availability; the status management unit specifically includes: The event-aware subunit is used to acquire data change events from the time-series data source side and extract the time information involved in the data change events; A positioning subunit, connected to the event sensing subunit, is used to locate at least one target data block affected by the data change event based on the time information; The state maintenance subunit, connected to the positioning subunit, is used to perform state update operations on each target data block; specifically, it is used to: increment the value of the first type of information and switch the second type of information to a state indicating that the cached data is unavailable.

[0016] A cache storage unit 14, connected to the state management unit, is used to cache data at the data block level and associate the state description information of the data block corresponding to each cached entry with the cached data; the cache storage unit specifically includes: The first storage subunit is used to cache data at the storage granularity of the data block, and to associate the status description information of the data block corresponding to the cached data with each cache entry; The second storage subunit is connected to the first storage subunit and is used to store cached data of data blocks in the first storage subunit whose access frequency is lower than a first preset threshold or whose storage time is greater than a second preset threshold. The version comparison subunit, connected to the first storage subunit, the second storage subunit, and the status management unit, is used to respond to a read request for any data block by obtaining the status description information of the associated record in the cache entry of the data block and comparing it with the status description information currently maintained by the status management unit for the data block. If the first type of information and the second type of information are consistent, the cache is determined to be hit and the cached data is returned from the corresponding storage subunit; if they are inconsistent, the cache is determined to be invalid and a cache invalidation indication is sent to the refresh control unit.

[0017] A refresh control unit 16, connected to the state management unit and the cache storage unit, is used to respond to a data read operation accessing any of the data blocks, determine whether the cached data of the data block is available based on the second type of information of the data block, and if it is unavailable, trigger a data reload operation for the data block; the refresh control unit specifically includes: The status detection subunit is used to respond to a data read operation, obtain the second type of information of the target data block, and determine whether the cached data of the target data block is available based on the second type of information; The overload execution subunit is connected to the state detection subunit and is used to obtain the data of the target data block from the time-series data source side when it is determined that the cached data is unavailable, write the obtained data into the cache storage unit, and switch the second type of information of the target data block to a state indicating that the cached data is available after the writing is completed. The concurrent arbitration subunit, connected to the state detection subunit and the reload execution subunit, is used to maintain a set of identity information for data blocks currently undergoing data reload operations. When the state detection subunit determines that the cached data of the target data block is unavailable, it queries the identity information set to see if the identity information of the target data block exists. If it exists, the current data read operation waits for the existing reload operation to complete. If it does not exist, the identity information of the target data block is added to the identity information set, and the reload execution subunit is triggered to perform the data reload operation. After the reload execution subunit completes its execution, the identity information of the target data block is removed from the identity information set. The refresh control unit further includes an incremental judgment subunit, which is connected to the state detection subunit and the overload execution subunit. When the state detection subunit determines that the cached data of the target data block is unavailable, it obtains the degree of difference between the current state description information of the target data block and the state description information of the associated record in the cache entry. If the degree of difference meets a preset condition, it obtains only the data within the time sub-interval of the change from the time-series data source side, merges it with the existing data in the cache, and writes it into the cache storage unit. If the degree of difference does not meet the preset condition, the overload execution subunit is triggered to perform a complete data overload operation.

[0018] A query processing unit 18, connected to the cache storage unit and the reporting system, is used to receive and parse data query instructions issued by the reporting system, break down the data query instructions into several sub-query tasks according to the time range involved, each sub-query task pointing to a corresponding data block, schedule the execution of each sub-query task, and merge the execution results of each sub-query task before feeding them back to the reporting system; the query processing unit specifically includes: The request parsing subunit is used to receive and parse the data query command issued by the reporting system, and extract the target time interval from the data query command. The task decomposition subunit, connected to the request parsing subunit, is used to call the partition management unit according to the target time interval and map the target time interval into a continuous data block sequence covering the interval; The data acquisition subunit is connected to the task decomposition subunit and is used to concurrently execute the cache reading operation of each data block in the data block sequence, and to merge the cached data of each data block or the data reloaded by the refresh control unit into the intermediate result set. An aggregation calculation subunit, connected to the data acquisition subunit, is used to perform aggregation operations on the intermediate result set according to the aggregation rules in the data query instruction; The result encapsulation subunit is connected to the aggregation calculation subunit and is used to encapsulate and return the results of the aggregation calculation in a format recognizable by the reporting system.

[0019] The following describes in detail the above-mentioned technical solutions of the present invention with reference to the specific circumstances of the time-series database interaction system for power generation enterprise reports in the embodiments of the present invention.

[0020] This invention proposes a time-series data interaction system between a power generation enterprise's reporting system and a real-time database. This system is deployed on a server within the power generation enterprise's information network and connects the reporting system and at least one real-time database via a communication network, as detailed below: I. System Overall Architecture This system adopts a layered and decoupled microservice architecture. The modules communicate with each other through well-defined interfaces. The system as a whole is divided into the following layers: Interface layer: Provides RESTful API interfaces for interacting with the reporting system, as well as adapter interfaces for interacting with the time series database.

[0021] Service layer: Includes query scheduling and result assembly modules, responsible for parsing, decomposing, scheduling and merging report requests.

[0022] Cache layer: Contains multi-level data caching modules, adopts a two-level storage architecture of memory and SSD, and caches data in units of time partitions.

[0023] Metadata layer: Includes data partitioning module and partition version vector manager, responsible for time partitioning, version vector maintenance and change monitoring.

[0024] Refresh Engine Layer: Contains a delayed-binding refresh engine, responsible for refreshing dirty partition data on demand.

[0025] II. Data Partitioning Module This module connects to a real-time database interface adapter and is used to divide time-series data into continuous and non-overlapping time partitions according to the time dimension. The size of the time partitions can be configured according to the data type and query mode, such as partitioning by hour, day, or month. This module generates a unique partition identifier for each time partition. Specifically, it includes: 1. Partition Granularity Configuration Subunit The partition granularity configuration subunit is used to configure the partition granularity corresponding to different data ages. The selection of partition granularity is based on the following principles: data backfilling usually occurs within the recent time window, so fine-grained partitioning can reduce the amount of data refreshed per time; historical data has a low probability of backfilling, so coarse-grained partitioning can reduce version vector maintenance overhead. Specifically, the partition granularity configuration subunit supports the following preset granularities and can be dynamically adjusted according to actual operation: Minute-level partitioning: The partition size is 15 minutes or 30 minutes, suitable for real-time data within the most recent hour; Hourly partitioning: The partition size is 1 hour, applicable to data within the most recent 24 hours; Day-level partitioning: The partition size is 1 day, suitable for data within 24 hours to 7 days; Weekly partitioning: The partition size is 1 week, suitable for data within 7 to 30 days; Monthly partitioning: The partition size is 1 month, suitable for historical data of more than 30 days.

[0026] The partition granularity configuration subunit persistently stores partition configuration information in a local configuration file, which is loaded when the system starts and can be dynamically modified through the management interface during runtime.

[0027] 2. Sub-unit for calculating partition boundaries This sub-unit is used to calculate the boundary of the time partition to which the given timestamp belongs, based on the given timestamp and the current partition granularity. The partition boundary calculation uses the following algorithm: Input: timestamp, partition granularity; Output: Partition start time, partition end time; set up: For the entered timestamp, For partition granularity, This is the partition start time. If the partition end time is given, then the partition boundary calculation formula is: ; ; in, This is for floor function. When When it is exactly an integer multiple of the partition boundary, Belongs to Partition for the start time; when When it is not an integer multiple, It belongs to the time partition corresponding to its floor.

[0028] For example, assuming the partition granularity is 3600 seconds, the timestamp =1705314180000, corresponding to 2025-01-15 14:23:00, then the calculation yields... This corresponds to 14:00:00 on 2025-01-15. This corresponds to 2025-01-15 15:00:00.

[0029] Meanwhile, the formula for calculating the partition list when querying a time range is: For query time range The set of partition start times covering this range is: ; Among them, the initial partition boundary , , To ensure that the floor function is performed, the partition containing the termination time is covered; To query the start timestamp, This is the query end timestamp.

[0030] The partition boundary calculation subunit also provides a batch calculation interface, which, given a time range and partition granularity, returns a list of boundaries for all partitions covering that time range.

[0031] 3. Partition Index Maintenance Subunit This subunit is responsible for maintaining an ordered partition index structure in memory, used to quickly locate the partition to which any timestamp belongs. The partition index uses a skip list or a balanced binary tree (such as a red-black tree) data structure, with the partition start time as the key and the partition metadata as the value.

[0032] When a new partition is created, the partition index maintenance sub-unit inserts the new partition's metadata into the index structure; when partitions are merged or split, the index structure is updated synchronously. The index structure supports the following operations: A. Find the partition based on the timestamp: complexity O(log N), where N is the total number of partitions; B. Find all covered partitions based on the time range: first locate the partition where the start timestamp is located, then sequentially traverse to the partition where the end timestamp is located; C. Get the minimum and maximum partitions: used for boundary judgment.

[0033] III. Partition Version Vector Manager The partition version vector manager connects to the data partitioning module. This manager maintains a version vector for each time partition to accurately track changes in the data within that partition.

[0034] 1. The version vector data structure includes the following fields: A. Partition Identifier: Uniquely identifies a time partition; B. Version Serial Number: A monotonically increasing integer that automatically increments whenever data within this partition changes, such as data backfilling, data modification, or data deletion. C. Partition Status Flag: Used to indicate whether the partition is currently in a clean, dirty, or refreshed state; D. Data Hash Value: A lightweight hash check value for the data within this partition, used to quickly verify data integrity.

[0035] 2. Change the monitoring subunit Used to acquire data change events in a real-time database, the change monitoring subunit supports the following two monitoring modes for different types of time-series databases: Mode 1: CDC (Change Data Capture) Mode For time-series databases that support CDC functionality, such as those supporting WAL log parsing, the change monitoring subunit registers a change listener through the CDC interface connected to the database. When data is written, updated, or deleted in the database, the CDC interface automatically pushes change events. Each change event contains the following information: change type (INSERT / UPDATE / DELETE), affected measurement point ID, timestamp of the changed data, and values ​​before and after the change.

[0036] Mode 2: Polling Mode For time-series databases that do not support CDC functionality, the change monitoring subunit adopts a timestamp-based polling method. Specifically, this subunit maintains a polling waterline and periodically (e.g., every 5 seconds) queries the real-time database for the number of records with timestamps greater than the waterline. If a new record is found, its metadata (timestamp range, measurement point distribution, etc.) is obtained, and a change event is constructed.

[0037] The implementation process of the polling pattern is as follows: A. The initial value of the water level is the system startup time minus a safety offset, such as 1 hour; B. After each poll, update the maximum timestamp found to the new watermark. C. To avoid missing data backfilling (writing data with timestamps less than the watermark), the polling mode is also combined with periodic full scans (e.g., once a day) for supplementary detection.

[0038] Regardless of the mode used, the change monitoring subunit ultimately outputs a standardized change event. The data structure includes: a unique event identifier, a change event type, a list of affected measurement point IDs, the minimum timestamp of the change data, the maximum timestamp of the change data, and the event occurrence time.

[0039] 3. Version Update Subunit Used to receive change events output by the change monitoring subunit, and execute the following processing flow: Step 1: Based on the minimum and maximum timestamps of the changed data in the change event, call the partition boundary calculation subunit of the data partitioning module to determine the list of affected time partitions. Since the change event may span multiple partitions, such as the time range of the changed data covering multiple hourly partitions, it is necessary to calculate the complete coverage area.

[0040] Step 2: Iterate through each partition in the list of affected time partitions and perform a version update operation on that partition: A. Read the current version serial number and increment it: New version serial number = current version serial number + 1; B. Set the partition status to dirty; C. Update the timestamp of the last version update to the current timestamp; D. Set the CRC64 hash value of the partition data to 0 (indicating that the hash value is invalid and will be recalculated after a refresh). E. Write the updated version vector to the metadata store.

[0041] Step 3: Record the version update event in the audit log. The log content includes: event ID, affected partition, old version serial number, new version serial number, and update time. The audit log can be used for troubleshooting and performance analysis.

[0042] The version update subunit uses an optimistic locking mechanism to ensure concurrency safety. When multiple change events affect the same partition at the same time, the version update subunit checks for version serial number conflicts. If a conflict occurs, the update operation is retried.

[0043] 4. Version query interface The version query interface allows other modules to query version information for a specified partition. The interface supports the following query methods: A. Single partition query: Input a partition identifier and return the complete version vector of that partition.

[0044] B. Time Range Query: Enter a time range and return a list of version vectors for all partitions covering that range.

[0045] C. Dirty Partition Query: Returns a list of all partitions currently in a dirty state, used for monitoring and debugging.

[0046] The version query interface also provides batch query optimization: when you need to query version information for multiple partitions, you can submit a list of partition identifiers at once, and the interface will return the corresponding version vector mapping table, reducing the network overhead caused by multiple calls.

[0047] IV. Delayed Binding Refresh Engine Connecting the partition version vector manager and the multi-level data caching module, unlike existing technologies that perform refreshes immediately or periodically after data changes, the delayed binding refresh engine in this embodiment of the invention adopts a strategy of separating marking and refreshing: when a data backfill event occurs, the partition version vector manager only marks the state of the affected partition as dirty, and the refresh operation is delayed until the actual query occurs. It includes the following sub-units: 1. Refresh trigger sub-unit When a report query request needs to access a specific partition, it is responsible for checking the status of that partition and deciding whether to trigger a refresh. Its workflow is as follows: Step 1: Receive the list of partition identifiers from the query scheduling module; Step 2: For each partition, call the version query interface of the partition version vector manager to obtain the current status of the partition; Step 3: If the partition is clean, no refresh is needed, and cached data can be used directly; Step 4: If the partition status is dirty, proceed to the refresh process. First, check if a refresh task for that partition already exists in the refresh deduplication sub-unit: If it exists, register the current query as a waiter for this refresh task; If it does not exist, create a new refresh task and make the current query the first waiter.

[0048] Step 5: If the partition status is "Refreshing", it means that the partition is being refreshed by other queries. The current query will wait for the refresh task to complete. Step 6: After all partition statuses have been confirmed, return the list of refresh tasks that need to be waited for to the query and scheduling module.

[0049] 2. Refresh the execution subunit The refresh execution subunit is responsible for actually performing the data refresh operation. Each refresh task runs independently, and its execution flow is as follows: Step 1: Obtain the partition identifier that needs to be refreshed and the partition start timestamp and partition end timestamp from its version vector; Step 2: Update the partition status from dirty to refresh and update the metadata storage synchronously. The timing of this status change is very important - after the status changes to refresh, other queries will no longer attempt to create duplicate refresh tasks. Step 3: Send a data query request to the time series database through the real-time database interface adapter. The query conditions are: A. Time range: partition boundaries; B. Measurement point range: Determined based on the specific requirements of the report query. If it is a general pre-load, then query all measurement points within this partition. C. Data precision: Raw sampled values, without aggregation.

[0050] Furthermore, to improve query efficiency, the refresh execution subunit of this embodiment supports the following optimization measures: A. Batch Query: Combines queries for multiple test points into a single batch request; B. Paging Reading: For partitions with large amounts of data, paging is used to read data in batches to avoid memory overflow; C. Compressed Transmission: Enable data compression to reduce network transmission volume.

[0051] Step 4: Receive the data result set returned by the time series database and organize it into cached data blocks according to the time order and measurement point order; Step 5: Write the data block to the multi-level data cache module, and record the version serial number corresponding to the data block at the same time; Step 6: After the refresh is complete, recalculate the CRC64 hash value of the partition. The calculation method is as follows: serialize all data records in the partition in chronological order and then calculate the CRC64 value. Step 7: Update the partition status from refresh to clean, keep the version serial number unchanged, update the timestamp of the last version update to the refresh completion time, and update the CRC64 hash value to the newly calculated value; Step 8: Notify all queries waiting for this refresh task (stored in the refresh deduplication sub-unit) that the refresh task has been completed and the data is available; Step 9: Remove the refresh task from the refresh deduplication sub-unit to release resources.

[0052] 3. Refresh the deduplication sub-units This sub-unit is a key component of the lazy-bound refresh engine, used to solve the problem of repeated refreshes when multiple concurrent queries hit the same dirty partition. Internally, it maintains a mapping table, denoted as the refresh task mapping table. The key of this mapping table is the partition identifier, and the value is the refresh task handle. The refresh task handle is an asynchronous task object that encapsulates the asynchronous execution result of the refresh execution sub-unit.

[0053] When the refresh trigger subunit needs to create a refresh task for a certain partition, the following logic is executed: First, the refresh task mapping table is synchronously locked to prevent concurrent modifications by multiple threads; Secondly, check if a refresh task handle for this partition already exists in the refresh task mapping table: If it already exists, it means that another query has already triggered a refresh task for this partition. The current query does not need to trigger it again, and directly returns the handle of the existing refresh task. If it does not exist, a new asynchronous refresh task will be created, which will call the refresh execution subunit to perform the actual data refresh operation.

[0054] Then, the newly created refresh task handle is stored in the refresh task mapping table, with the partition identifier as the key; Finally, after the newly created refresh task is completed, a cleanup operation is automatically triggered: the refresh task mapping table is re-synchronized and locked, the entry corresponding to the partition is removed from the mapping table, and resources are released.

[0055] Through the above mechanism, when multiple queries simultaneously hit the same dirty partition, the first query triggers the creation of a refresh task. Subsequent queries directly retrieve and wait for the results of the existing refresh task, thus ensuring that only one refresh operation is performed and all queries share the same refresh result. Furthermore, the refresh deduplication subunit also supports timeout control: if the execution time of a refresh task exceeds a preset threshold, such as 30 seconds, the task is automatically marked as timed out and removed from the refresh task mapping table, allowing subsequent queries to re-trigger the refresh task and preventing all related queries from permanently waiting due to a single refresh task being stuck.

[0056] 4. Incremental refresh judgment sub-unit To optimize the refresh strategy: when a partition is dirty, a full refresh of the entire partition is not necessarily required. This sub-unit analyzes the timestamp distribution of change events to determine whether an incremental refresh can be performed. Specifically, the incremental refresh determination sub-unit executes the following logic: Step 1: Obtain the version serial number of the current cached data for this partition. and the latest version serial number ; Step 2: If This indicates that only one change has occurred. Further, the time range of the change is determined: if the change range covers the entire partition, such as a full partition data update, a full refresh is triggered; if the change range only covers a sub-interval within the partition, such as 15-minute data backfilling within an hourly partition, an incremental refresh is triggered—only the data of that sub-interval is re-acquired and merged with the remaining data in the cache. Step 3: If This indicates that multiple changes have occurred. To avoid the complexity of multiple incremental refreshes, a full refresh is triggered directly.

[0057] The incremental refresh judgment subunit passes the refresh type (full or incremental) and the time range to be refreshed (for incremental refresh) to the refresh execution subunit.

[0058] V. Multi-level data caching module Connected to a delay-bound refresh engine, this module employs a tiered storage architecture. It stores data in time-partitioned units, with each cached data block associated with the version serial number of its corresponding partition.

[0059] 1. Tiered storage architecture A multi-level data caching module must contain at least the following two storage levels: The first layer, the L1 cache layer, is based on memory storage and offers the fastest read and write speeds, making it suitable for frequently accessed hot data. The L1 cache uses an LRU (Least Recently Used) eviction policy, and its capacity is configurable. Data in the L1 cache is organized at the partition level, with each partition's data being an independent cache entry in the L1 cache.

[0060] The second layer, the L2 cache layer, is based on solid-state drive storage, offering faster read and write speeds and is suitable for warm data. The L2 cache employs an LFU (Least Frequently Used) eviction policy, and its capacity is configurable. Data in the L2 cache is also organized at the partition level, but it supports finer-grained sub-partitions for incremental refresh.

[0061] The third layer, the L3 cache layer, is optional. For scenarios with extremely large volumes of historical data, an L3 cache layer can be configured, based on hard disk drives or object storage, and is suitable for cold data. The L3 cache mainly serves as a fallback guarantee for query performance.

[0062] 2. Cache data structure Each cache entry corresponds to a time partition, and its data structure includes the following fields: A. Partition Identifier: Uniquely identifies a time partition; B. Version Serial Number: Records the version serial number corresponding to this cached data; C. Actual data: Mapping structure, the outer key is the timestamp, the inner key is the measurement point identifier, and the value is the measurement point data; D. Cache write time: Records the timestamp of the cache entry being written; E. Access Count: Records the number of times this cache entry has been accessed; F. Last access time: Records the timestamp of the last time this cached entry was accessed; G. Data size: Records the number of bytes occupied by this cache entry.

[0063] 3. Cache operation interface The multi-level data caching module provides the following caching operation interfaces: First, the read interface. This interface accepts the partition identifier and the expected version serial number as input parameters and returns cached data. If the partition does not exist in the cache, it returns an empty value; if the partition exists in the cache but the version serial number does not match, it returns an empty value and records the version expiration log information.

[0064] Second, the write interface. This interface accepts the partition identifier, version serial number, data content, and target storage level as input parameters, and writes the data to the specified level. Before writing, it checks whether the partition already exists. If it exists and the newly written version serial number is updated, the original data is overwritten; if the newly written version serial number is older, the write operation is ignored.

[0065] Third, the eviction interface. This interface accepts the partition identifier as an input parameter and actively evictions the cache of the specified partition, used for cleanup operations during partition merging or splitting.

[0066] Fourth, the warm-up interface. This interface receives a list of partition identifiers and the target storage tier as input parameters, loads the data of the specified partition list into the target cache tier, and is used for rapid warm-up after system startup or fault recovery.

[0067] Fifth, the statistics interface. This interface has no input parameters and returns statistical information such as cache hit rate, capacity utilization of each level, and access frequency of each partition.

[0068] 4. Version comparison and expiration determination When a report query reads cached data, the multi-level data caching module executes the following version comparison logic: First, the corresponding cache entry is searched in the cache based on the partition identifier. If the cache entry does not exist, it is determined that the cache is missing, and a missing identifier is returned. If the cache entry exists, the version serial number recorded in the cache entry is compared with the current version serial number returned by the partition version vector manager. If they are equal, it is determined that the cache has been hit, and the cached data is returned. If they are not equal, it is determined that the cache has expired, and log information about the version inconsistency (including the partition identifier, cache version serial number, and current version serial number) is recorded, and an expiration identifier is returned.

[0069] After the cache expires, the delayed binding refresh engine is invoked to trigger an on-demand refresh. To avoid frequent version expiration logs consuming too much storage space, a sampling recording strategy can be adopted, for example, recording a new log every one hundred expiration events.

[0070] 5. Cache preheating mechanism To optimize report query performance, the system performs a cache preheating operation before the expected daily report generation time. Specifically: The system starts a preheating task based on the expected query time output by the report mode learning module, or according to a configured fixed time (e.g., 00:50 daily). The preheating task reads data from the real-time database in batches within the time partition to be queried, and writes it to a multi-level data cache module. During writing, the version sequence number of each cached partition is recorded as the current version sequence number in the partition version vector manager. The preheating task supports parallel execution and can preheat multiple partitions simultaneously. The concurrency of the preheating task can be configured according to system resources.

[0071] VI. Query Scheduling and Result Assembly Module It connects the reporting system and the multi-level data caching module, and is used to receive report query requests, split them into subqueries by time partition, schedule the execution of each subquery, and assemble the results of each subquery in chronological order before returning them to the reporting system.

[0072] 1. Query and parse sub-units This is used to receive HTTP requests sent by the reporting system, parse the query parameters in the request body, and the data structure of the report query request includes the following fields: start time, end time, measurement point list (a list of measurement point identifiers to be queried, supporting wildcards), aggregation type (values ​​include original value, average value, maximum value, minimum value, cumulative value, and count), aggregation window, filter conditions, sorting method, and maximum number of records.

[0073] And perform the following verification operations: A. Time range validity check: The start time must be less than the end time, and the time range cannot exceed the maximum query range configured by the system.

[0074] B. Measurement point list verification: The number of measurement points in the measurement point list cannot exceed the upper limit configured by the system, and each measurement point identifier must conform to the naming convention.

[0075] C. Aggregation parameter validation: The aggregation window must be a positive integer or zero, and the aggregation type must be one of the types supported by the system.

[0076] 2. Divide into sub-units by region Based on the time range queried, the partition boundary calculation subunit of the data partitioning module is invoked to decompose the time range into a list of consecutive time partitions. The decomposition algorithm is as follows (input parameters are the query start time and query end time, and the current partition granularity): The first step is to calculate the starting boundary of the partition where the start time is located; The second step is to calculate the end boundary of the partition where the end time is located. The calculation formula is as follows: ; The third step is to start from the initial partition boundary and generate the start and end times of each partition sequentially, with the partition granularity as the step size, until the end partition boundary is reached. The fourth step is to generate a partition identifier for each partition in the format of table_start_end_time; Fifth step, return the list of partition identifiers.

[0077] For example, if the query time range is from 14:23:00 to 15:47:00 and the partition granularity is 1 hour, then it will be split into two partitions: the 14:00-15:00 partition and the 15:00-16:00 partition.

[0078] 3. Cache query subunit The cache query subunit is responsible for querying data from each partition in parallel. For each partition in the partition list, the following operations are performed: The first step is to call the version query interface of the partition version vector manager to obtain the current version serial number of the partition; The second step is to call the read interface of the multi-level data caching module to try to obtain the cached data of the partition; Third, if the cache hits and the version matches, add the data of that partition to the result buffer; Fourth, if the cache misses or the version does not match, mark the partition as needing to be refreshed and add it to the list of partitions to be refreshed; The fifth step is to submit the list of partitions to be refreshed to the delayed binding refresh engine and wait for the refresh to complete. Step 6: Once all partition data is ready, arrange the data in the result buffer in chronological order and pass it to the aggregation calculation subunit.

[0079] To improve query efficiency, the cache query subunit adopts the following optimization methods: A. Parallel query: Uses a thread pool to query multiple partitions in parallel. The thread pool size is configurable and defaults to twice the number of CPU cores. B. Asynchronous waiting: For partitions that need to be refreshed, an asynchronous waiting mechanism is used to avoid blocking queries on other partitions; C. Timeout Control: Set the total query timeout. After the timeout, return the partially retrieved data and record a warning log.

[0080] 4. Aggregate Calculation Subunit The aggregation calculation subunit performs aggregation calculations on the query results based on the aggregation parameters in the query request. Supported aggregation types include: A. Raw Value Aggregation: Returns the original sampled data directly without aggregation. The number of records to be returned must be limited to the maximum number of records, and the data must be sorted according to the sorting method.

[0081] B. Average Aggregation: Groups data by aggregation window and calculates the average value of each measurement point within each window. For cases where window boundaries are incomplete, such as the start and end windows of a time range, only the actual data within the window is calculated.

[0082] C. Maximum value aggregation: Calculates the maximum value of each measuring point within each window according to the aggregation window.

[0083] D. Minimum value aggregation: Calculates the minimum value of each measuring point within each window according to the aggregation window.

[0084] E. Cumulative Value Aggregation: Calculates the cumulative value of each measurement point within each window according to the aggregation window. It is applicable to cumulative measurement points.

[0085] F. Count Aggregation: Calculates the number of records for each measurement point within each window according to the aggregation window.

[0086] The aggregation calculation subunit adopts a streaming computing mode: it reads data from each partition one by one, groups it by time window, and maintains the aggregation state within the window, including cumulative sum, maximum value, minimum value, count, etc., avoiding loading all data into memory before calculation. For large-scale data queries, such as queries spanning multiple days and thousands of measurement points, this streaming computing can significantly reduce memory usage.

[0087] 5. Result Encapsulation Subunit The result encapsulation subunit encapsulates the output of the aggregation calculation subunit into a format recognizable by the reporting system. Supported output formats include: JSON format, suitable for RESTful API calls, with a response structure containing three parts: return code, data body, and message; CSV format, suitable for data export, with the first row being the column header, including timestamps and names of each measurement point, followed by data rows; and Arrow columnar format, suitable for large data volume transmission, employing a columnar storage format, offering high compression ratio and fast parsing speed.

[0088] This subunit also supports response compression: gzip compression is automatically enabled when the response size exceeds a preset threshold.

[0089] VII. System Startup and Initialization Process The following initialization process is executed when this system starts up: Step 1: Load the configuration file, including partition granularity configuration, cache level configuration, time series database connection parameters, service port configuration, etc. Step 2: Connect to the time series database, verify the connection validity, and obtain the version and capability information of the time series database, such as whether it supports change data capture, etc. Step 3: Call the data partitioning module to create an initial time partition based on the current time and partition granularity configuration. The initial partition covers seven days before the system startup time to seven days after the system startup time, ensuring that the partition already exists when querying the most recent data. Step 4: Initialize the partition version vector manager, create a version vector for each initial partition, initialize the version serial number to zero, and set the status to clean; Step 5: Initialize the multi-level data caching module, allocate L1 and L2 cache capacity, and load the persistent cache index; Step 6: Start the delayed binding refresh engine and initialize the refresh thread pool. The size of the thread pool is configured according to system resources, and the default value is the number of CPU cores. Step 7: Start the change monitoring sub-unit of the partition version vector manager, select the change data capture mode or polling mode according to the capabilities of the time series database, and start monitoring data changes; Step 8: Start the HTTP service of the query scheduling and result assembly module to listen for query requests from the reporting system.

[0090] VIII. Partition Adaptive Adjustment Mechanism Preferably, this system also includes a partition adaptive adjustment subunit, used to dynamically adjust the partition granularity based on actual operating conditions, further optimizing performance. The partition adaptive adjustment subunit periodically performs the following analysis, for example, once daily: 1. Analyze the following metrics for each partition from the audit logs: Change frequency: the number of times the partition undergoes data changes per unit time; Query frequency: the number of times the partition is queried per unit time; Refresh time: the average time taken for the partition to be refreshed in the most recent few times; Data volume: the number of records and bytes in the partition.

[0091] 2. For partitions with high change frequency and long refresh time, determine whether they need to be split into finer-grained partitions. For example, if a certain hourly partition frequently experiences data backfilling, more than 10 times a day, and each refresh takes more than 500 milliseconds, then this partition should be split into four 15-minute partitions.

[0092] 3. For adjacent partitions with extremely low query frequency and small data volume, determine whether they need to be merged. For example, a group of daily partitions that have not been queried for 7 consecutive days, and whose data volume is less than the threshold, should be merged into a weekly partition.

[0093] 4. When performing partition adjustments, the partition index, version vector, and cache need to be updated synchronously. The adjustment process adopts a double buffering mechanism: first, a new partition structure is created, data is copied from the old partition to the new partition, the index pointer is switched after verification, and finally, the old partition resources are cleaned up.

[0094] In summary, the complete workflow of this system is as follows: Initialization phase: The data partitioning module creates time partitions, and the partition version vector manager initializes a version vector for each partition; Normal write phase: The real-time database receives on-site data. If it is normal real-time data, that is, the timestamp is close to the current time, it is written directly without triggering a version change. Data backfilling phase: When delayed data arrives, the monitoring sub-unit captures the event, locates the affected partition, increments the partition's version serial number, and sets its status to dirty. There is no data reading operation during this phase. Report Query Phase: When a report query arrives, the query scheduling module breaks it down into partitioned subqueries. For each partition, the caching module compares the version serial number. If they match, the cache is returned directly; otherwise, the delayed binding refresh engine is invoked.

[0095] On-demand refresh phase: The refresh engine checks the refresh deduplication mapping table. If there is no refresh task, it creates and executes it, refreshing only the data in that partition; if there is a refresh task, it waits for the shared results. Result return phase: After all partition data is ready, the aggregation calculation module performs aggregation, and the result encapsulation module formats and returns the results to the reporting system; Adaptive optimization phase: The partition adaptive adjustment subunit periodically analyzes operating indicators, dynamically adjusts the partition granularity, and continuously optimizes system performance.

[0096] Method Implementation Examples According to embodiments of the present invention, a time-series database interaction method for power generation enterprise reports is provided. Figure 2 This is a flowchart of a time-series database interaction method for power generation enterprise reports according to an embodiment of the present invention, as follows: Figure 2 As shown, the time-series database interaction method for power generation enterprise reports according to an embodiment of the present invention specifically includes: Step S201: The partition management unit divides the time series data into several continuous and non-overlapping data blocks according to the preset time granularity, and generates a unique identifier for each data block. Step S202: The status management unit maintains corresponding status description information for each data block; wherein, the status description information includes a first type of information reflecting the degree of data change and a second type of information reflecting data availability. Step S203: Cache data at the data block level using the cache storage unit, and associate the status description information of the data block corresponding to the cached data with each cache entry; Step S204: The query processing unit receives and parses the data query instruction issued by the reporting system, and breaks down the data query instruction into several sub-query tasks according to the time range involved. Each sub-query task points to the corresponding data block. Step S205: The refresh control unit responds to the data reading operation of each sub-query task on the corresponding data block by refreshing the data block, and determines whether the cached data of the data block is available based on the second type of information of the data block. If it is not available, a data reload operation for the data block is triggered. Step S206: The query processing unit schedules the execution of each sub-query task and merges the execution results of each sub-query task before feeding them back to the reporting system.

[0097] The embodiments of the present invention are method embodiments corresponding to the system embodiments described above. The specific operations of each step can be understood by referring to the description of the system embodiments, and will not be repeated here.

[0098] In summary, this invention proposes a time-series data interaction system between a power generation enterprise reporting system and a real-time database, including a data partitioning module, a partition version vector manager, a delayed binding refresh engine, a multi-level data caching module, and a query scheduling and result assembly module. This invention maintains a version vector containing a version sequence number for each time partition. When data backfilling occurs, it only increments the version number and marks the partition as dirty without performing a refresh, delaying the refresh operation until a query occurs, and only refreshing the affected partition data. Furthermore, this invention uses a refresh deduplication mechanism to allow multiple concurrent queries to share the same refresh result, reduces data read volume through incremental refresh judgment, and optimizes partition granularity through adaptive partition adjustment. Therefore, this invention solves the technical problems of difficult cache data consistency maintenance and excessive overhead of full refresh in data backfilling scenarios, and has advantages such as low processing overhead, on-demand refresh, concurrent sharing, high cache hit rate, and guaranteed data consistency.

[0099] Device Example 1 This invention provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it performs the steps described in the method embodiment.

[0100] Device Example 2 This invention provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor, performs the steps described in the method embodiment.

[0101] The computer-readable storage media described in this embodiment include, but are not limited to, ROM, RAM, disk, or optical disk.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A time-series database interaction system for power generation enterprise reports, characterized in that, include: The partition management unit is used to divide time-series data into several continuous and non-overlapping data blocks according to a preset time granularity, and generate a unique identifier for each data block. A status management unit, connected to the partition management unit, is used to maintain corresponding status description information for each data block; wherein, the status description information includes a first type of information reflecting the degree of data change and a second type of information reflecting data availability; A cache storage unit, connected to the state management unit, is used to cache data at the data block level and associate the state description information of the data block corresponding to the cached data with each cache entry. A refresh control unit, connected to the state management unit and the cache storage unit, is used to respond to a data read operation accessing any of the data blocks, determine whether the cached data of the data block is available based on the second type of information of the data block, and if it is not available, trigger a data reload operation for the data block. The query processing unit, connected to the cache storage unit and the reporting system, is used to receive and parse the data query instructions issued by the reporting system, break down the data query instructions into several sub-query tasks according to the time range involved, each sub-query task points to a corresponding data block, schedule the execution of each sub-query task, and merge the execution results of each sub-query task and feed them back to the reporting system.

2. The system according to claim 1, characterized in that, The partition management unit specifically includes: The granularity configuration subunit is used to configure the currently used time granularity; A boundary calculation subunit, connected to the granularity configuration subunit, is used to obtain the time granularity and calculate the boundary information of the data block to which the time stamp to be processed belongs based on the timestamp to be processed and the time granularity. An index maintenance subunit, connected to the boundary calculation subunit, is used to obtain the boundary information and construct an ordered index using the starting boundary in the boundary information as the key; wherein, the ordered index is used to locate the corresponding data block according to any timestamp, and to locate all data blocks covering any time range according to any time range.

3. The system according to claim 1, characterized in that, The status management unit specifically includes: The event-aware subunit is used to acquire data change events from the time-series data source side and extract the time information involved in the data change events; A positioning subunit, connected to the event sensing subunit, is used to locate at least one target data block affected by the data change event based on the time information; The state maintenance subunit, connected to the positioning subunit, is used to perform state update operations on each target data block; specifically, it is used to: increment the value of the first type of information and switch the second type of information to a state indicating that the cached data is unavailable.

4. The system according to claim 1, characterized in that, The cache storage unit specifically includes: The first storage subunit is used to cache data at the storage granularity of the data block, and to associate the status description information of the data block corresponding to the cached data with each cache entry; The second storage subunit is connected to the first storage subunit and is used to store cached data of data blocks in the first storage subunit whose access frequency is lower than a first preset threshold or whose storage time is greater than a second preset threshold. The version comparison subunit, connected to the first storage subunit, the second storage subunit, and the status management unit, is used to respond to a read request for any data block by obtaining the status description information of the associated record in the cache entry of the data block and comparing it with the status description information currently maintained by the status management unit for the data block. If the first type of information and the second type of information are consistent, the cache is determined to be hit and the cached data is returned from the corresponding storage subunit; if they are inconsistent, the cache is determined to be invalid and a cache invalidation indication is sent to the refresh control unit.

5. The system according to claim 1, characterized in that, The refresh control unit specifically includes: The status detection subunit is used to respond to a data read operation, obtain the second type of information of the target data block, and determine whether the cached data of the target data block is available based on the second type of information; The overload execution subunit is connected to the state detection subunit and is used to obtain the data of the target data block from the time-series data source side when it is determined that the cached data is unavailable, write the obtained data into the cache storage unit, and switch the second type of information of the target data block to a state indicating that the cached data is available after the writing is completed. The concurrent arbitration subunit, connected to the state detection subunit and the reload execution subunit, is used to maintain a set of identity information for data blocks currently undergoing data reload operations. When the state detection subunit determines that the cached data of the target data block is unavailable, it queries the identity information set to see if the identity information of the target data block exists. If it exists, the current data read operation waits for the existing reload operation to complete. If it does not exist, the identity information of the target data block is added to the identity information set, and the reload execution subunit is triggered to perform the data reload operation. After the reload execution subunit completes its execution, the identity information of the target data block is removed from the identity information set.

6. The system according to claim 1, characterized in that, The query processing unit specifically includes: The request parsing subunit is used to receive and parse the data query instruction issued by the reporting system, and extract the target time interval from the data query instruction; The task decomposition subunit, connected to the request parsing subunit, is used to call the partition management unit according to the target time interval and map the target time interval into a continuous data block sequence covering the interval; The data acquisition subunit is connected to the task decomposition subunit and is used to concurrently execute the cache reading operation of each data block in the data block sequence, and to merge the cached data of each data block or the data reloaded by the refresh control unit into the intermediate result set. An aggregation calculation subunit, connected to the data acquisition subunit, is used to perform aggregation operations on the intermediate result set according to the aggregation rules in the data query instruction; The result encapsulation subunit is connected to the aggregation calculation subunit and is used to encapsulate and return the results of the aggregation calculation in a format recognizable by the reporting system.

7. The system according to claim 5, characterized in that, The refresh control unit further includes an incremental judgment subunit, which is connected to the state detection subunit and the overload execution subunit. When the state detection subunit determines that the cached data of the target data block is unavailable, it obtains the degree of difference between the current state description information of the target data block and the state description information of the associated record in the cache entry. If the degree of difference meets a preset condition, it obtains only the data within the time sub-interval of the change from the time-series data source side, merges it with the existing data in the cache, and writes it into the cache storage unit. If the degree of difference does not meet the preset condition, the overload execution subunit is triggered to perform a complete data overload operation.

8. A time-series database interaction method for power generation enterprise reports, characterized in that, include: The partition management unit divides the time series data into several continuous and non-overlapping data blocks according to the preset time granularity, and generates a unique identifier for each data block. The status management unit maintains corresponding status description information for each data block; wherein, the status description information includes a first type of information reflecting the degree of data change and a second type of information reflecting data availability; The cache storage unit caches data at the data block level, and records the status description information of the data block corresponding to the cached data for each cache entry. The query processing unit receives and parses the data query instructions issued by the reporting system, and breaks down the data query instructions into several sub-query tasks according to the time range involved. Each sub-query task points to the corresponding data block. By refreshing the control unit in response to the data reading operation of each sub-query task on the corresponding data block, the cached data of the data block is determined according to the second type of information of the data block. If it is not available, a data reload operation for the data block is triggered. The query processing unit schedules the execution of each sub-query task and merges the execution results of each sub-query task before feeding them back to the reporting system.

9. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the time-series database interaction method for power generation enterprise reports as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an implementation program for information transmission, which, when executed by a processor, implements the steps of the time-series database interaction method for power generation enterprise reports as described in claim 8.

Images