Time series data collection method and device based on dynamic time window and fingerprint deduplication

Abstract

The application relates to a time series data collection method and device based on a dynamic time window and fingerprint deduplication. The method comprises the following steps: establishing a sliding time window with a variable length based on a current system time, and only querying incremental time series data in the window; adaptively scaling the window by real-time calculation of the ratio of the collected data quantity to the expected quantity; setting a trigger interval as half of the length of the sliding time window to form time overlap, so as to capture out-of-order data; extracting key features from each time series data and generating data fingerprints; and performing preliminary screening by using a Bloom filter and accurate deduplication in a key-value storage system. Compared with an existing fixed polling scheme, the technology can dynamically adjust resources according to the data flow rate, reduce system load and improve real-time performance, the overlapping window guarantees data integrity, the two-level fingerprint deduplication ensures zero-repeated collection, and the technology is suitable for an efficient incremental synchronization scene of an InfluxDB or other non-transaction log time series database.

Description

Technical Field

[0001] This application relates to the field of data synchronization technology, and in particular to an InfluxDB time-series data acquisition method, apparatus, computer-readable storage medium, and electronic device based on dynamic time windows and fingerprint deduplication. Background Technology

[0002] With the rapid convergence of the Internet of Things (IoT) and the Industrial Internet, scenarios such as equipment monitoring and business metric analysis place higher demands on the efficient storage and real-time processing of massive amounts of time-series data. Time-series databases (TSDBs), as specialized databases adapted to the characteristics of time-series data (such as time correlation, high-frequency writes, and dynamic growth), have been widely used in this field. InfluxDB, as a typical example, has become the mainstream choice for scenarios such as industrial IoT and DevOps monitoring due to its high compression ratio, high write throughput, and time-series-optimized storage structure and query performance.

[0003] However, in real-time data acquisition scenarios, current time-series databases (typically InfluxDB) face significant technical bottlenecks: unlike traditional relational databases (such as MySQL) which use transaction logs (such as binlog) to support Change Data Capture (CDC) technology to achieve low-latency data synchronization, InfluxDB lacks a native transaction log mechanism, making it difficult to achieve efficient real-time data synchronization through existing CDC technology. This is particularly prominent in scenarios that require rapid response to data changes (such as real-time alerts and real-time data analysis).

[0004] To address the above issues, existing technologies have proposed various incremental data acquisition schemes, but all of them have significant limitations: 1. Scheduled Full Table Scan Solution: This method involves periodically performing a full table scan to retrieve data changes. While simple to implement, it is extremely resource-intensive. In scenarios where data updates are infrequent (i.e., when data is sparse), this solution can lead to a significant waste of computing, storage, and network resources due to repeatedly scanning large amounts of unchanged data.

[0005] 2. Fixed-interval polling scheme: This scheme queries newly added data at preset fixed time intervals (e.g., every 5 seconds). While this reduces resource consumption compared to full polling, it cannot adapt to dynamic changes in data rate. When the data write rate suddenly increases (e.g., high-frequency data reporting during device malfunction alarms), the fixed interval may lead to a significant increase in data acquisition latency, resulting in insufficient real-time performance.

[0006] 3. Timestamp-based incremental data acquisition scheme: This scheme relies on predefined timestamp fields (such as timestamp) in data records to filter new data and determines the data range by comparing the current query time with the last query time. However, this scheme has two drawbacks: First, if the timestamp field is missing or incorrect when data is written (e.g., due to asynchronous device clocks), incremental data cannot be accurately identified; second, when network fluctuations or asynchronous device reporting cause data to arrive out of order (i.e., the timestamp of subsequent data is earlier than the timestamp of already collected data), this scheme cannot effectively handle out-of-order data, which may lead to data loss or duplication.

[0007] In summary, the existing technology has the following main shortcomings: Serious waste of resources: Full polling or fixed-interval polling generates a large number of invalid queries in data-sparse scenarios, resulting in inefficient use of computing resources, storage resources and network bandwidth; Insufficient real-time performance: The fixed polling interval cannot be dynamically adjusted according to the data write rate, making it difficult to meet the real-time synchronization requirements in scenarios with sudden data surges; Risk of duplicate data collection: In short-cycle polling scenarios, if the data writing and collection operations overlap in time (such as triggering collection immediately after batch writing on the device), the same data may be collected multiple times. Weak ability to handle out-of-order data: Existing solutions generally rely on the assumption that timestamps monotonically increase, and lack an effective mechanism for handling out-of-order data caused by network latency, asynchronous reporting by devices, etc. in real-world scenarios, which affects data integrity and consistency.

[0008] Therefore, there is an urgent need to develop a new time-series data acquisition method that can adaptively adjust the acquisition window according to the data flow rate while taking into account deduplication and out-of-order tolerance. Summary of the Invention

[0009] To address the aforementioned issues, this application proposes a novel method and apparatus for real-time acquisition of InfluxDB time-series data based on dynamic time windows and fingerprint deduplication. The method and apparatus of this application can achieve low-latency and high-reliability acquisition of InfluxDB time-series data without increasing the system load, while avoiding duplicate data acquisition.

[0010] To achieve the above objectives, the present invention mainly adopts the following technical strategies: I. Dynamic Time Window Polling Mechanism This mechanism balances the real-time nature of data collection with resource efficiency by adaptively adjusting the query time range and trigger frequency. Specifically, it includes: 1. Sliding time window controls the query range: Each time data collection is triggered, the query time window is defined as [current time - currentWindowSize, current time] (currentWindowSize is a dynamically adjustable parameter) based on the current system time. The window slides forward with the system time, covering only the most recent potential new data.

[0011] 2. Dynamic window adjustment based on data rate: The window size is dynamically scaled by calculating the fill rate (fetchedRecords / expectedRecords) in real time: When data surges (fill rate is higher than the threshold), the window shrinks to a minimum of 10 seconds to improve real-time performance; when data is sparse (fill rate is lower than the threshold), the window expands to a maximum of 5 minutes to reduce load.

[0012] 3. Overlapping window sampling design: The interval for triggering the collection task is set to currentWindowSize / 2 to ensure that there is some overlap in the time range of two adjacent queries (overlap duration is currentWindowSize / 2). The historical out-of-order data caused by network latency and other reasons is captured through the subsequent overlapping window.

[0013] II. Fingerprint Deduplication Mechanism This mechanism achieves efficient deduplication of massive time-series data by generating unique data identifiers and combining them with multi-level verification. Specifically, it includes: 1. Unique Hash Fingerprint Generation: Extract key features from time-series data (such as timestamps, field values, tag information, etc.), concatenate them into a unique fingerprint string according to preset rules, perform SHA-256 hash calculation on it, and generate a fixed-length hash value as the data fingerprint.

[0014] 2. Two-level deduplication using Bloom filter and Redis: A two-level verification strategy is adopted: First, the fingerprint is quickly screened for existence using a Bloom filter (high space efficiency); fingerprints that are determined to be "possible" by the Bloom filter are further precisely verified by Redis (high concurrency read and write support), ultimately achieving efficient deduplication of massive amounts of data.

[0015] To achieve the above strategy, this application adopts the following specific measures: (1) Adaptive window adjustment algorithm: The window size is adjusted in real time based on the data fill rate (the ratio of the actual data volume to the expected data volume), breaking through the traditional fixed window polling mode. When the actual data volume exceeds 120% of the expected volume, the window shrinks by 20%; when the actual data volume is less than 80% of the expected volume, the window expands by 20%, effectively dealing with the problem of fluctuation in time-series data rate.

[0016] (2) Overlapping window out-of-order processing model: The overlapping window sampling design is adopted, and the trigger interval is set to 1 / 2 of the window size, so that the time range of two adjacent queries overlaps by 50%, ensuring that out-of-order data caused by network latency and other reasons can be captured by subsequent windows, avoiding data loss caused by time gaps in traditional solutions.

[0017] (3) Data fingerprint generation: Data fingerprints are generated through a standardized concatenation process (timestamp nanosecond conversion, field value sorting, and tag information arranged in lexicographical order) combined with the SHA-256 hash algorithm. The standardized concatenation rules eliminate the impact of differences in field order, and the hash algorithm ensures the uniqueness of the fingerprint.

[0018] (4) Two-level deduplication design of Bloom filter + Redis: The high efficiency of Bloom filter is used to quickly screen duplicate data, and then Redis is used for secondary confirmation. The two complement each other to form an efficient and accurate deduplication mechanism to ensure that duplicate data is effectively identified and filtered.

[0019] Specifically, this application provides the following technical solutions: The first aspect of this application provides a time-series data acquisition method based on dynamic time windows and fingerprint deduplication, the method comprising: S1. Based on the current system time, establish a sliding time window of variable length, query only the incremental time series data within the window, and slide the window forward as the system time progresses; S2. After each polling, based on the ratio of the amount of data collected this time to the expected amount of data, the length of the sliding time window is adaptively scaled in real time to balance the real-time performance of data collection and the system load; S3. Set the trigger interval between two adjacent polls to half the current length of the sliding time window, so that the adjacent windows partially overlap in time, in order to capture out-of-order data; S4. Extract key features from each time-series data and generate a data fingerprint; S5. Use a Bloom filter to perform initial screening of the data fingerprints, and then use a key-value storage system to accurately deduplicate the fingerprints that were initially detected, filtering out duplicate data to ensure that the same fingerprint is collected only once.

[0020] Furthermore, in the method of this application, the initial length of the sliding time window in step S1 is 60 seconds, and its length is limited to a range of a minimum of 10 seconds and a maximum of 5 minutes.

[0021] Furthermore, in the method of this application, step S2 involves dynamically and adaptively scaling the length of the sliding time window based on the ratio of the currently collected data volume to the expected data volume. The rule for this real-time adaptive scaling is as follows: When the ratio of the data volume to the expected data volume is higher than 120%, the window length will be reduced by 20% (minimum of 10 seconds). When the ratio of the data volume to the expected data volume is less than 80%, the window length will be increased by 20% (not exceeding 5 minutes).

[0022] Furthermore, in the method of this application, the key features described in step S4 include: Timestamps, expressed with nanosecond precision; Field values ​​are sorted by field name and then concatenated; floating-point numbers are retained to 6 decimal places. Tag key-value pairs are concatenated after being sorted lexicographically by TagKey.

[0023] Furthermore, in the method of this application, step S4 also includes: S41. Extract key features from each time series data point; S42. Standardize and concatenate the fingerprint string according to preset rules; S43. Perform SHA-256 hash calculation on the fingerprint string to generate a fixed-length hash value as a data fingerprint (unique fingerprint).

[0024] Furthermore, in the method of this application, the key-value storage system mentioned in step S5 is Redis, and the fingerprint is stored in the form of set members, and a time-to-live (TTL) is set to control memory usage.

[0025] Furthermore, in the method of this application, step S5 also includes: S51. Use a Bloom filter to perform preliminary deduplication verification on the data fingerprint. If the Bloom filter determines that the fingerprint does not exist (returns false), then the data is not duplicated, and the data is identified as new data for further processing. S52. If the Bloom filter determines that the fingerprint may exist (returns true), then connect to Redis via Jedis and check whether the fingerprint exists in the specified Redis set; S53. If it exists, return true to indicate that the data is duplicated and skip the data; if it does not exist, return false to consider the data as new data and proceed with subsequent processing.

[0026] A second aspect of this application provides a time-series data acquisition device based on dynamic time windows and fingerprint deduplication, the device comprising: The sliding time window creation module is used to create a sliding time window of variable length based on the current system time. Only the incremental time series data within the window is queried, and the window slides forward with the system time. An adaptive window scaling module is used to adaptively scale the length of the sliding time window in real time after each polling based on the ratio of the amount of data collected this time to the expected amount of data, so as to balance the real-time performance of data collection and the system load. The trigger interval setting module is used to set the trigger interval between two adjacent polls to half the current length of the sliding time window, so that the adjacent windows partially overlap in time, which is used to capture out-of-order data. The data fingerprint generation module is used to extract key features from each time-series data and generate a data fingerprint; The fingerprint deduplication module is used to perform initial screening of the data fingerprints using a Bloom filter, and then to perform precise deduplication of the fingerprints matched in the initial screening using a key-value storage system, filtering duplicate data and ensuring that the same fingerprint is collected only once.

[0027] The device implements the steps of the aforementioned time-series data acquisition method based on dynamic time windows and fingerprint deduplication during operation.

[0028] A third aspect of this application provides an electronic device, including: a memory and a processor; Memory: Used to store computer programs; Processor: Used to execute the computer program to implement the steps of the aforementioned time-series data acquisition method based on dynamic time windows and fingerprint deduplication.

[0029] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the aforementioned time-series data acquisition method based on dynamic time windows and fingerprint deduplication.

[0030] In summary, compared with the prior art, the method of this application has the following significant advantages: (1) Adaptive to data rate fluctuations: The dynamic time window polling mechanism balances the real-time nature of data acquisition with the consumption of system resources by adjusting the query window size in real time. It can adapt to the dynamic changes of time-series data rate, effectively reduce invalid queries, and reduce system load by more than 40% compared with traditional solutions.

[0031] (2) Out-of-order data processing: The window overlap sampling design is adopted. By partially overlapping the adjacent query time ranges, historical out-of-order data caused by network delays and other reasons can be captured. This solves the problem of out-of-order data that is difficult to handle by traditional solutions to a certain extent and improves the integrity of data collection.

[0032] (3) Efficient deduplication: A standardized fingerprint generation process is defined for the InfluxDB data structure. Combined with the two-level deduplication mechanism of fast initial screening by Bloom filter and accurate verification by Redis, efficient and accurate deduplication of massive time series data is achieved, avoiding the waste of storage and processing resources due to duplicate data.

[0033] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings involved in the description of the embodiments will be briefly introduced below. It should be noted that the accompanying drawings only show some embodiments of this application. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating the overall implementation of the time-series data acquisition method based on dynamic time windows and fingerprint deduplication in this application.

[0036] Figure 2 This is a schematic diagram of fixed-interval polling with no overlap.

[0037] Figure 3 This is a schematic diagram of overlapping sliding windows in the method of this application.

[0038] Figure 4 This is a flowchart of the two-stage deduplication process using Bloom filter and Redis in the method described in this application.

[0039] Figure 5 This is a structural diagram of the time-series data acquisition device based on dynamic time windows and fingerprint deduplication in this application.

[0040] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0042] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0043] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0044] Figure 1 The diagram shows the overall implementation flow of the time-series data acquisition method based on dynamic time windows and fingerprint deduplication provided in this application, including the following steps: S1. Based on the current system time, establish a sliding time window of variable length, query only the incremental time series data within the window, and slide the window forward as the system time progresses; S2. After each polling, based on the ratio of the amount of data collected this time to the expected amount of data, the length of the sliding time window is adaptively scaled in real time to balance the real-time performance of data collection and the system load; S3. Set the trigger interval between two adjacent polls to half the current length of the sliding time window, so that the adjacent windows partially overlap in time, in order to capture out-of-order data; S4. Extract key features from each time-series data and generate a data fingerprint; S5. Use a Bloom filter to perform initial screening of the data fingerprints, and then use a key-value storage system to accurately deduplicate the fingerprints that were initially detected, filtering out duplicate data to ensure that the same fingerprint is collected only once.

[0045] To more clearly illustrate the technical solution of this application, the following will provide further explanation through specific scenario embodiments.

[0046] To implement the above method, this application provides a time-series data acquisition device based on dynamic time windows and fingerprint deduplication, including the following functional modules: (a) Source Data Acquisition Module The core purpose of the source data acquisition module is: (1) Solving the problem of time-series data rate fluctuation It adapts to the burstiness (a surge in data volume within a short period of time, such as batch reporting by devices) and sparsity (a sudden drop in data volume within a long period of time, such as during off-peak hours at night) of InfluxDB data writes, avoiding resource waste or data delays caused by fixed-interval polling.

[0047] (2) Balancing real-time performance with system load During peak data periods, the window is automatically reduced to increase the collection frequency (minimum 10 seconds), while during off-peak periods, the window is expanded to reduce the number of queries (maximum 5 minutes), achieving optimal resource allocation.

[0048] (3) Ensure the integrity of out-of-order data By using an overlapping window sampling design, out-of-order data can be captured by subsequent windows, avoiding data loss caused by time gaps in traditional solutions.

[0049] To achieve the above objectives, the present invention employs a dynamic time window polling mechanism, wherein the time window represents the time range for the query.

[0050] The dynamic time window polling mechanism automatically adjusts the size of subsequent query time windows by monitoring the data arrival rate in real time. The system maintains a variable sliding time window (initial value 60 seconds), and only queries newly added data within the window range during each polling.

[0051] The dynamic time window polling mechanism works as follows: (1) Initialization phase The first step is to initialize the time window and determine the boundary range of the window size currentWindowSize. In this solution, the initial window size is set to 60 seconds, and the boundary range is limited to: the minimum window size is 10 seconds and the maximum window size is 5 minutes.

[0052] The adjustment factor for the dynamic time window size is 0.2.

[0053] Boundary condition handling: Minimum interval protection: When the window is reduced to 10 seconds, the interval is no less than 5 seconds (to avoid high-frequency queries overwhelming the database).

[0054] Maximum interval limit: When the window is expanded to 5 minutes, the interval shall not exceed 150 seconds (to ensure the minimum acquisition frequency).

[0055] (2) Polling trigger In the polling triggering phase, there are two important design ideas: the sliding window model and the overlap sampling guarantee design.

[0056] Sliding window model: Assuming the current window size is currentWindowSize, each time a query is triggered, data within the range of [current time - currentWindowSize, current time] is retrieved, and the window slides forward with the system time.

[0057] Overlap sampling guarantee: In this scheme, the timer triggers a collection task every currentWindowSize / 2 time intervals to ensure that there is some overlap in the time range of adjacent queries (the duplicate collection data in the overlapping range is efficiently deduplicated by the fingerprint deduplication mechanism module).

[0058] (3) Data collection Based on the sliding window model of this scheme, the start and end times of the current time window are calculated respectively: start = current time – currentWindowSize; end = current time; The SQL statement to execute the InfluxDB query is: SELECT * FROM measurement WHERE time>=start AND time <end; Here, 'time' is a built-in field in InfluxDB used to record the time when the data was generated.

[0059] After the data collection and query SQL is executed, it is necessary to count the number of records fetchedRecords retrieved in this transaction to adjust the size of the subsequent query window.

[0060] (4) Window adjustment In this scheme, the window size is dynamically adjusted according to the data arrival rate. Specifically, the time window size is dynamically adjusted according to the following rules: When the amount of data collected in a single session exceeds 120% of the expected value, the subsequent query window will be reduced by 20%. When the amount of data collected in a single session is less than 80% of the expected value, the subsequent query window will be expanded by 20%. The query window size is limited to a preset range (10 seconds to 5 minutes).

[0061] In addition, window adjustment mainly relies on dynamic window adjustment algorithms.

[0062] Timing of adjustment: Adjustment is performed after each polling cycle ends.

[0063] Specifically, the core of the dynamic window adjustment algorithm is to predict the expected amount of data based on historical data, and use this as the basis for adjusting the window size in real time. The data arrival rate is calculated by the ratio of the current amount of data queried to the expected amount of data (such as 1,000 records), and then the size of the next window is adjusted in real time, thus breaking through the traditional fixed window polling mode.

[0064] double expectedRecords = getExpectedRecords(); / / Predict based on historical data double ratio = fetchedRecords / expectedRecords; if (ratio>1.2) { / / Data too dense currentWindowSize = Math.max( minWindowSize, (long)(currentWindowSize * (1 - adjustFactor)) ); } else if (ratio<0.8) { / / Data sparsity currentWindowSize = Math.min( maxWindowSize, (long)(currentWindowSize * (1 + adjustFactor)) ); } Compared to a fixed polling interval, this invention introduces a dynamic adjustment algorithm that adapts to the data rate. By calculating the fill rate (fetchedRecords / expectedRecords) in real time, the window size is dynamically scaled. When data surges, the window is automatically reduced (minimum 10 seconds) to improve real-time performance, and when data is sparse, the window is expanded (maximum 5 minutes) to reduce the load.

[0065] Furthermore, in actual production processes, InfluxDB often generates out-of-order data due to factors such as equipment re-reporting historical data after a network outage and network latency. For example... Figure 2 As shown, during fixed-interval polling, the design of non-overlapping time ranges leads to the inability to process out-of-order data, resulting in permanent loss.

[0066] To address the potential for out-of-order data, this solution employs an overlapping window sampling design. Each query covers the latter half of the previous window, and subsequent overlapping windows capture historical out-of-order data. This approach can mitigate the out-of-order data issue to some extent and improve the completeness of data collection. Table 1 and Figure 3 The image shows an example of a window range (assuming an initial window duration of 60 seconds).

[0067] Table 1. Examples of Dynamic Time Window Adjustment

[0068] (ii) Fingerprint deduplication mechanism module The core purpose of the fingerprint deduplication mechanism module is: (1) Solve the problem of duplicate data collection caused by short-cycle polling When performing dynamic time window overlap queries, the same data may be captured by multiple windows (e.g., data is queried at least twice under a 50% overlap rate).

[0069] (2) Avoid network retransmission or duplicate reporting of services Handle duplicate data packets generated in scenarios such as device disconnection and reconnection, and abnormal retries.

[0070] (3) Ensure data idempotency This ensures that the downstream system processes the same data only once, regardless of how many times it is collected, thus guaranteeing that the data is only written to the target end once.

[0071] By concatenating key features of data (such as timestamps, field values, tag information, etc.) into a unique fingerprint string according to specific rules, and then performing a hash calculation (such as SHA-256) on it to generate a fixed-length hash value (i.e., fingerprint). In massive datasets, if two data sets have completely identical fingerprint strings (their hash values ​​are the same after hash calculation), they can be considered duplicate data.

[0072] 1. Components of a fingerprint Table 2 Examples of Data Fingerprint Standardization Processing Rules

[0073] 2. Fingerprint generation process (1) Timestamp standardization The time stored in InfuxDB is generally in UTC time format, which needs to be standardized to nanosecond precision.

[0074] Original timestamp: 1659984005000 (milliseconds) → Converted to nanoseconds: 1659984005000000000.

[0075] (2) Field value normalization processing Sort by field name to avoid hash inconsistencies caused by out-of-order sorting. Concatenate fields in the form of field name=field value. Separate multiple fields with "|". Floating-point numbers retain 6 decimal places to eliminate misjudgments caused by differences in floating-point number storage.

[0076] (3) Add Tag information Tag keys are sorted lexicographically, such as device_id=SN123, region=CN, and are separated by the concatenation separator "|TAG|", which distinguishes them from field data.

[0077] (4) Construct fingerprint string The timestamp, field value, and tag information are concatenated using the "|" symbol to form a fingerprint string.

[0078] Suppose the following data points are retrieved from InfluxDB: { "timestamp": 1659984005000, / / Millisecond-level timestamp "measurement": "sensor_data", "tags": { "device_id": "SN123", "region": "CN" }, "fields": { Temperature: 26.5 "humidity": 45.2, "status": "normal" } } The fingerprint string obtained after processing according to the above standardization rules is in the following form: 1659984005000000000|humidity=45.200000|status=normal|temperature=26.500000|TAG|device_id=SN123|TAG|region=CN.

[0079] (5) Generate fingerprint identifier To ensure accurate, fast, and reliable deduplication, this solution uses the SHA-256 hash algorithm to map different data records to unique fingerprint identifiers, preventing collision attacks.

[0080] The input data (fingerprint string) is hashed using the SHA-256 algorithm to generate a fixed-length fingerprint string, which takes the following form: a591a6d40bf420404a011733cfb7b190d62c65bf0bcda32b57b277d9ad9f146e.

[0081] 3. Deduplication process In this scheme, in order to implement the dynamic time window polling mechanism, it is necessary to rely on a deduplication mechanism to remove duplicate data that may be generated in the overlapping window.

[0082] To efficiently deduplicate massive amounts of data and ensure that duplicate data is written only once on the target device, this solution employs a two-stage deduplication approach using a Bloom filter and Redis.

[0083] As an efficient data structure, Bloom filters have become a key component in deduplication of massive amounts of data due to their advantages such as high efficiency, speed, and low memory consumption. However, Bloom filters are prone to false positives, which may misclassify new data as duplicates (the false positive rate is usually 0.1%-1%).

[0084] Therefore, in order to achieve complete deduplication, this solution adopts a combination of Bloom filter and Redis for deduplication.

[0085] like Figure 4 As shown, the specific deduplication process is as follows: (1) Generate a fixed-length fingerprint string through the fingerprint generation process; (2) First, perform preliminary deduplication verification on the data fingerprint using a Bloom filter. If the Bloom filter considers that the fingerprint does not exist (returns false), then the data is not duplicated and is identified as new data, which can be processed later. (3) If the Bloom filter thinks the fingerprint may exist (returns true), then connect to Redis through Jedis (a Java language client library for operating Redis) and check whether the fingerprint exists in the specified Redis set (fixed key); (4) If it exists, return true to indicate that the data is duplicated and skip the data; if it does not exist, return false to indicate that the data is new and can be processed later.

[0086] This solution uses a Bloom filter for efficient initial screening and Redis for precise verification. By combining these methods, the memory pressure and data processing volume of Redis storage can be significantly reduced, achieving efficient deduplication of massive amounts of data.

[0087] This solution can also employ another combined deduplication scheme, the specific implementation method of which is as follows: (1) First, use the mightContain method of the BloomFilter object to judge the fingerprint. If it returns false, it is determined to be new data and is directly handed over to the downstream for processing.

[0088] (2) If true is returned, it may be duplicate data and needs to be further verified by Redis. Use Jedis's sismember method to check whether the data exists in the Redis Set data structure.

[0089] (3) If false is returned, it means that the same data was not found in Redis, and it is determined to be new data and directly handed over to the downstream for processing; if true is returned, it is determined to be duplicate data, and the data is skipped directly without processing.

[0090] Figure 5 The image shows a time-series data acquisition device based on dynamic time windows and fingerprint deduplication proposed in this application. The device includes: The sliding time window creation module is used to create a sliding time window of variable length based on the current system time. Only the incremental time series data within the window is queried, and the window slides forward with the system time. An adaptive window scaling module is used to adaptively scale the length of the sliding time window in real time after each polling based on the ratio of the amount of data collected this time to the expected amount of data, so as to balance the real-time performance of data collection and the system load. The trigger interval setting module is used to set the trigger interval between two adjacent polls to half the current length of the sliding time window, so that the adjacent windows partially overlap in time, which is used to capture out-of-order data. The data fingerprint generation module is used to extract key features from each time-series data and generate a data fingerprint; The fingerprint deduplication module is used to perform initial screening of the data fingerprints using a Bloom filter, and then to perform precise deduplication of the fingerprints matched in the initial screening using a key-value storage system, filtering duplicate data and ensuring that the same fingerprint is collected only once.

[0091] When the above-mentioned device is in operation, it implements the steps of the time-series data acquisition method based on dynamic time window and fingerprint deduplication disclosed in this application.

[0092] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of apparatus, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0093] like Figure 6 As shown in the illustration, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the aforementioned time-series data acquisition method based on dynamic time windows and fingerprint deduplication.

[0094] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0095] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types: Object-oriented programming languages, such as Java, Smalltalk, C++, etc. Conventional procedural programming languages, such as "C" or similar programming languages.

[0096] The execution methods of program code include, but are not limited to: It runs entirely on the user's computer; Part of it executes on the user's computer, and part of it executes on a remote computer; Execute as a standalone software package; It is executed entirely on a remote computer or server.

[0097] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers via an internet service provider, for example, by utilizing the internet.

[0098] Furthermore, this application also discloses a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the various steps of the time-series data acquisition method based on dynamic time windows and fingerprint deduplication disclosed in this application.

[0099] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following: (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0100] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0101] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0102] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0103] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0104] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0105] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0106] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0107] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0108] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0109] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0110] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program includes program code for executing the time-series data acquisition method based on dynamic time windows and fingerprint deduplication disclosed in this application. When the computer program is executed by a processing device, it can achieve the functions defined in the embodiments of this application.

[0111] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0112] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A time-series data acquisition method based on dynamic time windows and fingerprint deduplication, characterized in that, The time-series data is time-series data from a time-series database without transaction logs, and the method includes: S1. Based on the current system time, establish a sliding time window of variable length, query only the incremental time series data within the window, and slide the window forward as the system time progresses; S2. After each polling, based on the ratio of the amount of data collected this time to the expected amount of data, the length of the sliding time window is adaptively scaled in real time to balance the real-time performance of data collection and the system load; S3. Set the trigger interval between two adjacent polls to half the current length of the sliding time window, so that the adjacent windows partially overlap in time, in order to capture out-of-order data; S4. Extract key features from each time-series data and generate a data fingerprint; S5. Use a Bloom filter to perform initial screening of the data fingerprints, and then use a key-value storage system to accurately deduplicate the fingerprints that were initially detected, filtering out duplicate data to ensure that the same fingerprint is collected only once.

2. The method according to claim 1, characterized in that, The initial length of the sliding time window in step S1 is 60 seconds, and its length is limited to a minimum of 10 seconds and a maximum of 5 minutes.

3. The method according to claim 1, characterized in that, In step S2, the length of the sliding time window is adaptively scaled in real time based on the ratio of the amount of data collected this time to the expected amount of data. The rule for real-time adaptive scaling is as follows: When the ratio of the data volume to the expected data volume is higher than 120%, the window length will be reduced by 20%. When the ratio of the data volume to the expected data volume is less than 80%, increase the window length by 20%.

4. The method according to claim 1, characterized in that, The key features mentioned in step S4 include: Timestamps, expressed with nanosecond precision; Field values ​​are sorted by field name and then concatenated; floating-point numbers are retained to 6 decimal places. Tag key-value pairs are concatenated after being sorted lexicographically by TagKey.

5. The method according to claim 1, characterized in that, Step S4 also includes: S41. Extract key features from each time series data point; S42. Standardize and concatenate the fingerprint string according to preset rules; S43. Perform SHA-256 hash calculation on the fingerprint string to generate a fixed-length hash value as the data fingerprint.

6. The method according to claim 1, characterized in that, The key-value storage system mentioned in step S5 is Redis, and the fingerprint is stored as a set member, with a time-to-live (TTL) set to control memory usage.

7. The method according to claim 6, characterized in that, Step S5 also includes: S51. Use a Bloom filter to perform preliminary deduplication verification on the data fingerprint. If the Bloom filter determines that the fingerprint does not exist, the data is not duplicated, and the data is identified as new data for further processing. S52. If the Bloom filter determines that a fingerprint may exist, then connect to Redis via Jedis to check if the fingerprint exists in the specified Redis set; S53. If the data exists, skip it; if it does not exist, consider it new data and proceed with further processing.

8. A time-series data acquisition device based on dynamic time windows and fingerprint deduplication, characterized in that, The time-series data is time-series data from a time-series database without transaction logs, and the device includes: The sliding time window creation module is used to create a sliding time window of variable length based on the current system time. Only the incremental time series data within the window is queried, and the window slides forward with the system time. An adaptive window scaling module is used to adaptively scale the length of the sliding time window in real time after each polling based on the ratio of the amount of data collected this time to the expected amount of data, so as to balance the real-time performance of data collection and the system load. The trigger interval setting module is used to set the trigger interval between two adjacent polls to half the current length of the sliding time window, so that the adjacent windows partially overlap in time, which is used to capture out-of-order data. The data fingerprint generation module is used to extract key features from each time-series data and generate a data fingerprint; The fingerprint deduplication module is used to perform initial screening of the data fingerprints using a Bloom filter, and then to perform precise deduplication of the fingerprints matched in the initial screening using a key-value storage system, filtering duplicate data and ensuring that the same fingerprint is collected only once.

9. An electronic device, characterized in that, include: Memory and processor; Memory: Used to store computer programs; Processor: for executing the computer program to implement the steps of the time-series data acquisition method based on dynamic time window and fingerprint deduplication as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the time-series data acquisition method based on dynamic time windows and fingerprint deduplication as described in any one of claims 1-7.

Images