Time sequence data acquisition method and device based on dynamic time window and fingerprint deduplication

Abstract

The invention relates to a time series data acquisition method and device based on a dynamic time window and fingerprint deduplication. The method comprises the following steps: establishing a sliding time window with a variable length by taking the current system time as a reference, and only querying incremental time sequence data in the window; adaptively zooming the window by calculating the ratio of the collected data volume to the expected volume in real time; setting the trigger interval as half of the length of the sliding time window to form a time overlap, thereby capturing out-of-order data; key features of each piece of time series data are extracted, and data fingerprints are generated; a bloom filter is used for primary screening, and accurate duplicate removal is carried out in a key-value storage system; compared with an existing fixed polling scheme, the method has the advantages that resources can be dynamically adjusted along with data flow rate, system load is reduced, real-time performance is improved, data integrity is guaranteed by overlapped windows, zero repeated acquisition is guaranteed by fingerprint two-stage duplicate removal, and the method is applicable to efficient incremental synchronization scenes of transactional log-free time sequence databases such as InfluxDB and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data synchronization, and in particular relates to an InfluxDB time series data collection method and device based on a dynamic time window and fingerprint deduplication, a computer readable storage medium, and an electronic device. BACKGROUND

[0002] With the rapid integration of the Internet of Things (IoT) and the Industrial Internet, scenarios such as device monitoring and business index analysis have higher requirements for efficient storage and real-time processing of massive time series data. Time series databases (TSDB) are specialized databases that adapt to the characteristics of time series data, such as temporal correlation, high-frequency writing, and dynamic growth. They have been widely used in this field. InfluxDB, as a typical representative, has become the mainstream choice for industrial IoT and DevOps monitoring scenarios, thanks to its high compression ratio, high write throughput, and time series-oriented storage structure and query performance.

[0003] However, in real-time data collection scenarios, current time series databases (typified by InfluxDB) face significant technical bottlenecks. Unlike traditional relational databases (such as MySQL), which support change data capture (CDC) technology through transaction logs (such as binlog) to achieve low-latency data synchronization mechanisms, InfluxDB lacks native transaction log mechanisms, making it difficult to achieve efficient real-time data synchronization through existing CDC technology. This is particularly prominent in scenarios that require rapid response to data changes, such as real-time alerts and real-time data analysis.

[0004] To address the above problems, existing technologies have proposed various incremental data collection schemes, but all have significant limitations: 1. Timed full polling scheme: This method periodically performs full table scan queries to obtain data changes. Although this method is simple to implement, it consumes a large amount of resources. In scenarios where data updates are infrequent (i.e., when data is sparse), this method can cause significant waste of computing, storage, and network resources due to repeated scanning of large amounts of unchanged data.

[0005] 2. Fixed interval polling scheme: This method queries new data at a pre-set fixed time interval (e.g., every 5 seconds). Although this method reduces resource consumption compared to the full polling method, it cannot adapt to dynamic changes in data rates. When the data write rate suddenly increases (e.g., high-frequency data reporting during device abnormal alarm), the fixed interval may result in a significant increase in data collection delay, leading to insufficient real-time performance.

[0006] 3. Incremental collection scheme based on timestamp: Filter new data by a predefined timestamp field (such as timestamp) in data records, determine data range by comparing current query time with last query time. However, this scheme has two defects: first, if the timestamp field is missing or incorrect when data is written (such as device clock is not synchronized), it cannot accurately identify incremental data; second, when network fluctuation or device asynchronous reporting causes out-of-order data arrival (i.e. the timestamp of subsequent data is earlier than the timestamp of already collected data), this scheme cannot effectively handle out-of-order data, which may cause data loss or duplication.

[0007] In summary, the existing technology mainly has the following shortcomings: Serious waste of resources: full polling or fixed interval polling produces a large number of invalid queries in sparse data scenarios, resulting in inefficient use of computing resources, storage resources and network bandwidth; Real-time performance is insufficient: fixed polling interval cannot be dynamically adjusted according to data write rate, and it is difficult to meet real-time synchronization requirements in data burst scenarios; Risk of repeated collection: in short-period polling scenarios, if there is a time overlap between data writing and collection operations (such as device-side batch writing followed by immediate triggering of collection), the same data may be collected multiple times; Weak processing capability for out-of-order data: existing schemes generally rely on the assumption that timestamps are monotonically increasing, and lack effective processing mechanisms for out-of-order data caused by network delay, device asynchronous reporting, etc., affecting data integrity and consistency.

[0008] Therefore, it is urgent to develop a new time series data collection method that can adaptively adjust the collection window according to data flow rate, while taking into account deduplication and out-of-order tolerance. SUMMARY

[0009] To solve the above problems, the present application proposes a new InfluxDB time series data real-time collection method and device based on dynamic time window and fingerprint deduplication, which can realize low-latency and high-reliable collection of InfluxDB time series data without increasing system burden, while avoiding data duplication.

[0010] To achieve the above purpose, the present application mainly adopts the following technical strategies: I. Dynamic time window polling mechanism This mechanism balances the real-time performance and resource efficiency of data collection by adaptively adjusting the query time range and triggering frequency, which specifically includes: 1. Sliding time window control query range: each time the data collection is triggered, the current system time is taken as the benchmark to define the query time window as [current time - currentWindowSize, current time] (currentWindowSize is a dynamic adjustment parameter), the window slides forward with the system time, and only covers the potential newly added data in the recent period.

[0011] 2. Window dynamic adjustment based on data rate: dynamically scale the window size by real-time calculation of the fill rate (fetchedRecords / expectedRecords): when the data surge (fill rate is higher than the threshold), the window is reduced to the minimum of 10 seconds to improve real-time performance; when the data is sparse (fill rate is lower than the threshold), the window is expanded to the maximum of 5 minutes to reduce the load.

[0012] 3. Overlapping window sampling design: the interval of the collection task trigger is set to currentWindowSize / 2, ensuring that there is partial overlap (overlap duration is currentWindowSize / 2) between the time ranges of adjacent two queries, and capturing historical out-of-order data caused by network delay and other reasons through subsequent overlapping windows.

[0013] II. Fingerprint deduplication mechanism This mechanism realizes efficient deduplication of massive time series data by generating unique data identifiers and combining multi-level verification, specifically including: 1. Unique hash fingerprint generation: extract the key features of time series data (such as timestamp, field value, Tag information, etc.), concatenate them into a unique fingerprint string according to the preset rules, perform SHA-256 hash calculation on it, and generate a fixed-length hash value as the data fingerprint.

[0014] 2. Bloom filter + Redis two-level deduplication: adopt a two-level verification strategy: first, use Bloom filter to quickly screen the existence of fingerprints (high space efficiency); for fingerprints determined by Bloom filter as "possibly existing", further verify them accurately through Redis (high concurrent read-write support), and finally realize efficient deduplication of massive data.

[0015] In order to realize the above strategy, the following specific measures are adopted in the present application: (1) Adaptive window adjustment algorithm: based on the data fill rate (the ratio of actual data volume to expected data volume), the window size is adjusted in real time, breaking through the traditional fixed window polling mode. When the actual data volume exceeds 120% of the expected data volume, the window is reduced by 20%; when the actual data volume is less than 80% of the expected data volume, the window is expanded by 20%, effectively dealing with the problem of time series data rate fluctuation.

[0016] (2) Overlapping window out-of-order processing model: an overlapping window sampling design is adopted, the trigger interval is set to 1 / 2 of the window size, the time range of adjacent two queries is overlapped by 50%, ensuring that the out-of-order data caused by network delay and the like can be captured by the subsequent window, avoiding data loss caused by time gap in the traditional scheme.

[0017] (3) Data fingerprint generation: data fingerprints are generated through a standardized splicing process (timestamp nanosecond processing, field value sorting, and Tag information arranged in dictionary order) combined with a SHA-256 hash algorithm. The standardized splicing rule eliminates the influence of field order difference, and the hash algorithm ensures the uniqueness of the fingerprints.

[0018] (4) Bloom filter + Redis two-stage deduplication design: the Bloom filter is used for rapid preliminary screening of repeated data, and the Redis is used for secondary confirmation, and the two complement each other to form an efficient and accurate deduplication mechanism, ensuring that repeated data is effectively identified and filtered.

[0019] Specifically, the application provides the following technical solutions: The first aspect of the application provides a time series data collection method based on a dynamic time window and fingerprint deduplication, which comprises the following steps: S1. Taking the current system time as a reference, a sliding time window with variable length is established, only incremental time series data in the window is queried, and the window slides forward with the system time; S2. After each polling, the length of the sliding time window is scaled in real time and adaptively based on the ratio of the amount of data collected this time to the expected amount of data, so as to balance the real-time collection and system load; S3. The trigger interval of adjacent two polling is set to half of the current length of the sliding time window, so that adjacent windows are partially overlapped in time, and used for capturing out-of-order data; S4. Key features are extracted from each time series data and a data fingerprint is generated; S5. The data fingerprints are preliminarily screened by using a Bloom filter, and the fingerprints that pass the preliminary screening are accurately deduplicated by using a key-value storage system, so as to filter repeated data and ensure that the same fingerprint is collected only once.

[0020] Further, in the method of the application, the initial length of the sliding time window in step S1 is 60 seconds, and the length is limited in the range of 10 seconds to 5 minutes.

[0021] Further, in the method of the application, in step S2, the length of the sliding time window is scaled in real time and adaptively based on the ratio of the amount of data collected this time to the expected amount of data, wherein the real-time adaptive scaling rule is: When the ratio of the data volume to the expected data volume is higher than 120%, the window length is reduced by 20% (minimum not less than 10 seconds); When the ratio of the data volume to the expected data volume is lower than 80%, the window length is expanded by 20% (maximum not more than 5 minutes).

[0022] Further, in the method of the application, the key features in step S4 include: a timestamp expressed in nanosecond precision; a field value spliced after sorting by field name, with 6 decimal places reserved for floating point numbers; a Tag key-value pair spliced after sorting by TagKey dictionary order.

[0023] Further, in the method of the application, step S4 further includes: S41. Extracting key features from each time series data; S42. Standardizing and splicing into a fingerprint string according to a preset rule; S43. Performing SHA-256 hash calculation on the fingerprint string to generate a fixed-length hash value as a data fingerprint (unique fingerprint).

[0024] Further, in the method of the application, the key-value storage system in step S5 is Redis, and the fingerprint is stored in the form of a set member, and a time to live TTL is set to control memory occupation.

[0025] Further, in the method of the application, step S5 further includes: S51. Using a Bloom filter to perform preliminary deduplication verification on the data fingerprint, if the Bloom filter considers that the fingerprint does not exist (returns false), the data is not duplicated, and the data is identified as new data for subsequent processing; S52. If the Bloom filter considers that the fingerprint may exist (returns true), connect to Redis through Jedis to check whether the fingerprint exists in the specified Redis set; S53. If it exists, return true indicating that the data is duplicated, and skip the data; if it does not exist, return false, identifying the data as new data for subsequent processing.

[0026] The second aspect of the application provides a time series data collection device based on a dynamic time window and fingerprint deduplication, which comprises: a sliding time window creation module for establishing a sliding time window with variable length based on the current system time, querying only the incremental time series data within the window, and sliding the window forward with the system time; an adaptive window scaling module, configured to scale the length of the sliding time window in real time after each polling, based on the ratio of the amount of data collected this time and the expected amount of data, to balance the real-time performance of data collection and system load; a trigger interval setting module, configured to set the trigger interval between two adjacent polling as half of the current length of the sliding time window, so that adjacent windows partially overlap in time, for capturing out-of-order data; a data fingerprint generating module, configured to extract key features from each time-series data and generate a data fingerprint; a fingerprint deduplication module, configured to use a Bloom filter to preliminarily screen the data fingerprints, and then use a key-value storage system to accurately deduplicate the preliminarily screened fingerprints, filter out duplicate data, and ensure that the same fingerprint is collected only once.

[0027] The device realizes the steps of the aforementioned time-series data collection method based on dynamic time window and fingerprint deduplication when in operation.

[0028] The third aspect of the present application provides an electronic device, comprising a memory and a processor; the memory, configured to store a computer program; the processor, configured to execute the computer program to realize the steps of the aforementioned time-series data collection method based on dynamic time window and fingerprint deduplication.

[0029] The fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program, when executed by a processor, realizes the steps of the aforementioned time-series data collection method based on dynamic time window and fingerprint deduplication.

[0030] In summary, compared with the prior art, the method of the present application has the following significant advantages: (1) adaptive data rate fluctuation: the dynamic time window polling mechanism balances the real-time performance of data collection and system resource consumption by adjusting the query window size in real time, and can adapt to the dynamic change characteristics of time-series data rate, effectively reducing invalid queries, and reducing system load by more than 40% compared with traditional schemes.

[0031] (2) out-of-order data processing: the window overlap sampling design can capture historical out-of-order data caused by network delay and other reasons through the partial overlap of adjacent query time ranges, to a certain extent, solve the out-of-order problem that the traditional scheme cannot handle, and improve the integrity of data collection.

[0032] (3) Efficient deduplication: A standardized fingerprint generation process is defined for the InfluxDB data structure, combined with a two-level deduplication mechanism of Bloom filter rapid preliminary screening and Redis accurate verification, realizing efficient and accurate deduplication of massive time series data, and avoiding waste of storage and processing resources caused by repeated data.

[0033] Other features and advantages of the present application will be described in detail in the following description, or can be understood by implementing the related technical solutions of the present application. The purpose and other advantages of the present application can be achieved by the technical features and technical means explicitly indicated in the specification, claims and drawings, and obtained by the implementation process of these technical contents. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly explain the technical solutions of the embodiments of the present application, the drawings involved in the embodiment description will be briefly introduced as follows. It should be pointed out that the drawings only show part of the embodiments of the present application. For those skilled in the art, other related drawings can be derived from these drawings without creative labor.

[0035] Figure 1 The overall implementation flowchart of the time series data acquisition method based on dynamic time window and fingerprint deduplication of the present application.

[0036] Figure 2 The non-overlapping fixed interval polling schematic diagram.

[0037] Figure 3 The sliding window with overlap in the method of the present application.

[0038] Figure 4 The Bloom filter + Redis two-level deduplication flowchart in the method of the present application.

[0039] Figure 5 The composition structure diagram of the time series data acquisition device based on dynamic time window and fingerprint deduplication of the present application.

[0040] Figure 6 The structure schematic diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described clearly and completely in the following, combined with the drawings in the embodiments of the present application. It should be pointed out that the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0042] In this document, the terms "comprise" and "comprising" and any form of these terms, such as "comprises" and "comprising", are open-ended language that means "including but not limited to", i.e. an open-ended group of elements in which any additional element can be added. The term "based on" should be interpreted as "based at least in part on", i.e. the recited condition or factor can not be the only one that acts on the conclusion. The term "one embodiment" should be interpreted as "at least one embodiment", i.e. the described embodiment is not the only possible implementation, and other similar embodiments can exist.

[0043] In this application, the terms "one" and "multiple" used to modify the relevant elements or features are illustrative rather than restrictive. Unless otherwise explicitly stated in the context, "one" should be understood as "at least one", and "multiple" should be understood as "at least two". Those skilled in the art should reasonably interpret these terms according to the semantics and logical relationship of the context, ensuring that they cover the possibility of "one or more".

[0044] Figure 1 The overall implementation process of the time series data collection method based on dynamic time window and fingerprint deduplication provided by the present application is shown, which includes the following steps: S1. Establish a sliding time window with variable length based on the current system time, query only the incremental time series data within the window, and slide the window forward with the system time; S2. After each polling, based on the ratio of the amount of data collected this time to the expected amount of data, the length of the sliding time window is scaled in real time to balance the real-time collection and system load; S3. Set the trigger interval between adjacent polling to half the current length of the sliding time window, so that adjacent windows partially overlap in time to capture out-of-order data; S4. Extract key features from each time series data and generate data fingerprints; S5. Use Bloom filter to preliminarily screen the data fingerprints, and then use key-value storage system to accurately deduplicate the fingerprints that hit the preliminary screening, filter out duplicate data, and ensure that the same fingerprint is collected only once.

[0045] In order to more clearly illustrate the technical solutions of the present application, the following will be further described through specific examples of implementation.

[0046] In order to realize the above method, the present application provides a time series data collection device based on dynamic time window and fingerprint deduplication, which includes the following functional modules: (1) Source data collection module The core purpose of the source data collection module is: (1) Solve the time series data rate fluctuation problem Adapt to the burstiness of InfluxDB data writing (a large amount of data in a short period of time, such as batch reporting of devices) and sparsity (a large amount of data in a long period of time, such as low peak at night), avoid resource waste or data delay of fixed interval polling.

[0047] (2) Balance real-time and system load Automatically reduce the window to increase the collection frequency (minimum 10 seconds) during data peak, and expand the window to reduce the query times (maximum 5 minutes) during low valley, to realize optimal allocation of resources.

[0048] (3) Ensure the integrity of out-of-order data Through the overlapping window sampling design, it is ensured that the out-of-order data can be captured by the subsequent window, avoiding the data loss caused by the time gap in the traditional scheme.

[0049] In order to achieve the above purpose, a dynamic time window polling mechanism is adopted in the present application, wherein the time window represents the time range of the query.

[0050] The dynamic time window polling mechanism automatically adjusts the size of the subsequent query time window by monitoring the data arrival rate in real time. The system maintains a variable sliding time window (initial value 60 seconds), and only queries the newly added data in the window range each time polling.

[0051] The working process of the dynamic time window polling mechanism is as follows: (1) Initialization stage The initialization of the time window and the determination of the boundary range of the window size currentWindowSize are the first step. In the present scheme, the initial window size is set to 60 seconds, and the boundary range is limited: the minimum window is 10 seconds, and the maximum window is 5 minutes.

[0052] The adjustment factor adjustFactor of the dynamic time window size is 0.2.

[0053] Boundary condition processing: Minimum interval protection: when the window is reduced to 10 seconds, the interval is not less than 5 seconds (to avoid high-frequency query from crushing the database).

[0054] Maximum interval limit: when the window is expanded to 5 minutes, the interval is not more than 150 seconds (to ensure the minimum collection frequency).

[0055] (2) Polling trigger In the polling trigger stage, there are two important design ideas: sliding window model and overlapping sampling guarantee design.

[0056] Sliding window model: assuming the size of the current window is currentWindowSize, every time the query is triggered, the data in the range of [current time - currentWindowSize, current time] is fixedly obtained, and the window slides forward with the system time.

[0057] Overlapping sampling guarantee: in this scheme, the timer triggers the collection task every currentWindowSize / 2 time, ensuring that the time ranges of adjacent queries exist partially overlap (the repeated collection data in the range overlap part relies on the fingerprint deduplication mechanism module to realize efficient deduplication).

[0058] (3) Data collection According to the sliding window model of the present scheme, the start time and end time of the current time window are calculated respectively: start = current time - currentWindowSize; end = current time; The SQL for executing the InfluxDB query is constructed as: SELECT * FROM measurement WHERE time>=start AND time<end; Wherein, time is a built-in field in InfluxDB, used to record the data generation time.

[0059] After the execution of the data collection query SQL, the number of records fetched this time fetchedRecords needs to be counted, which is used to adjust the size of the subsequent query window.

[0060] (4) Window adjustment In this scheme, the window size is dynamically adjusted according to the data arrival rate. Specifically, the time window size is dynamically adjusted according to the following rules: When the amount of data collected at a time exceeds 120% of the expected value, the subsequent query window is reduced by 20%; When the amount of data collected at a time is less than 80% of the expected value, the subsequent query window is expanded by 20%; Wherein, the query window size is limited within the preset range (10 seconds-5 minutes).

[0061] In addition, window adjustment mainly relies on the window dynamic adjustment algorithm to realize.

[0062] Trigger adjustment occasion: adjust after each polling is completed.

[0063] Specifically, the core of the window dynamic adjustment algorithm is to predict the expected data volume based on historical data, which is used as the basis for real-time adjustment of window size. The data arrival rate is calculated by the ratio of the current queried data volume to the expected data volume (such as 1000), and then the size of the next window is adjusted in real time, thereby breaking the traditional fixed window polling mode.

[0064] double expectedRecords = getExpectedRecords(); / / Based on historical data prediction double ratio = fetchedRecords / expectedRecords; if (ratio>1.2) { / / Data is too dense currentWindowSize = Math.max( minWindowSize, (long)(currentWindowSize * (1 - adjustFactor)) ); } else if (ratio<0.8) { / / Data is sparse currentWindowSize = Math.min( maxWindowSize, (long)(currentWindowSize * (1 + adjustFactor)) ); } Compared with the fixed polling interval, the present application introduces a dynamic adjustment algorithm of data rate self-adaptation, which dynamically scales the window size by real-time calculation of the fill rate (fetchedRecords / expectedRecords). When the data is surging, the window is automatically reduced (minimum 10 seconds) to improve real-time performance, and when the data is sparse, the window is expanded (maximum 5 minutes) to reduce the load.

[0065] In addition, in the actual production process, due to the re-reporting of historical data after the device is disconnected from the network and network delay, InfluxDB often produces out-of-order data. As shown in Figure 2 When polling at a fixed interval, due to the design of no overlapping time range, out-of-order data cannot be processed, resulting in permanent loss.

[0066] In order to deal with the possible disordered data, this solution adopts the sampling design of overlapping windows. Each query will cover the second half of the data in the previous window. The subsequent overlapping windows are used to capture the historical disordered data, which can solve the disordered data problem to a certain extent and improve the integrity of data collection. Figure 3 Example window ranges are shown (assuming initial window = 60 seconds).

[0067] Table 1 Dynamic time window adjustment example

[0068] (2) Fingerprint deduplication mechanism module The core purpose of the fingerprint deduplication mechanism module is to: (1) Solve the problem of repeated collection caused by short-cycle polling When dynamic time windows are queried with overlapping queries, the same data may be captured by multiple windows (for example, data is queried at least twice with a 50% overlap rate).

[0069] (2) Avoid network retransmission or repeated reporting of services Processes duplicate data packets generated by scenarios such as device disconnection and reconnection, and abnormal retries.

[0070] (3) Ensuring data idempotence Ensure that no matter how many times the same data is collected, the downstream system only processes it once, thereby ensuring that the data is only written once to the target end.

[0071] This approach combines key data features (such as timestamps, field values, and tag information) into a unique fingerprint string according to specific rules. This is then hashed (e.g., using SHA-256) to generate a fixed-length hash value (the fingerprint). In a massive amount of data, if two data items have identical fingerprint strings (they have the same hash value after hashing), they are considered duplicates.

[0072] 1. Fingerprint components Table 2 Example of data fingerprint standardization processing rules

[0073] 2. Fingerprint generation process (1) Timestamp standardization The time stored in InfuxDB is generally in UTC time format and needs to be normalized to nanosecond precision.

[0074] Original timestamp: 1659984005000 (milliseconds) → Converted to nanoseconds: 16599840050000000000.

[0075] (2) Field value normalization Sort by field name to avoid hash inconsistency caused by disorder. Concatenate in the form of field name = field value. Multiple fields are separated by "|". Floating-point numbers retain 6 decimal places to eliminate misjudgments caused by floating-point number storage differences.

[0076] (3) Add Tag information Tag keys are sorted in lexicographical order, such as device_id=SN123, region=CN, and the delimiter "|TAG|" is used to distinguish them from field data.

[0077] (4) Constructing fingerprint string Use the "|" symbol to concatenate the timestamp, field value, and tag information to form a fingerprint string.

[0078] Assume that the following data points are queried from InfluxDB: { "timestamp": 1659984005000, / / millisecond timestamp "measurement": "sensor_data", "tags": { "device_id": "SN123", "region": "CN" }, "fields": { "temperature": 26.5, "humidity": 45.2, "status": "normal" } } The fingerprint string obtained after processing according to the above standardization rules is in the form of: 1659984005000000000|humidity=45.200000|status=normal|temperature=26.500000|TAG|device_id=SN123|TAG|region=CN.

[0079] (5) Generate fingerprint identification In order to ensure accurate, fast and reliable deduplication, this solution uses the SHA-256 hash algorithm to map different data records into unique fingerprint identifiers to prevent collision attacks.

[0080] The input data (fingerprint string) is hashed using the SHA-256 algorithm to generate a fixed-length fingerprint string, which has the following form: a591a6d40bf420404a011733cfb7b190d62c65bf0bcda32b57b277d9ad9f146e.

[0081] 3. De-duplication process In this scheme, in order to realize the dynamic time window polling mechanism, it is necessary to rely on the de-duplication mechanism to de-duplicate the possible repeated data in the overlapping window.

[0082] In order to efficiently de-duplicate massive data, the target end only writes repeated data once. In this scheme, a two-level de-duplication method of Bloom filter + Redis is adopted.

[0083] As a kind of efficient data structure, Bloom filter has become a key component in massive data de-duplication due to its advantages such as high efficiency, fast speed and low memory occupation. However, Bloom filter has false positive, which may misjudge new data as repeated (the false positive rate is usually 0.1%-1%).

[0084] Therefore, in order to realize complete de-duplication, this scheme adopts the combined de-duplication method of Bloom filter + Redis.

[0085] As shown in Figure 4 , the specific de-duplication process is as follows: (1) Generate a fixed-length fingerprint string through the fingerprint generation process; (2) First, check the data fingerprint through the Bloom filter for preliminary de-duplication. If the Bloom filter considers that the fingerprint does not exist (returns false), the data is not repeated, and the data is identified as new data, which can be processed subsequently; (3) If the Bloom filter considers that the fingerprint may exist (returns true), connect to Redis through Jedis (a client library for operating Redis in Java) to check whether the fingerprint exists in the specified Redis set (fixed key); (4) If it exists, return true, indicating that the data is repeated, and skip this data; if it does not exist, return false, indicating that the data is new, which can be processed subsequently.

[0086] This scheme uses Bloom filter for efficient preliminary screening and Redis for accurate verification. Through the above combined method, the memory pressure and data processing amount of Redis storage can be significantly reduced, and efficient de-duplication of massive data is realized.

[0087] The scheme can also use another combination deduplication scheme, and the specific implementation method is: (1) First, use the mightContain method of the BloomFilter object to judge the fingerprint, if it returns false, it is determined to be new data, and then directly hand it over to the downstream processing.

[0088] (2) If it returns true, it may be duplicate data, which needs to be handed over to Redis for further accurate verification. The sismember method of Jedis is used to check whether the data exists in the Set data structure of Redis.

[0089] (3) If it returns false, it means that the same data is not found in Redis, and it is determined to be new data, which is directly handed over to the downstream processing; if it returns true, it is determined to be duplicate data, which is directly skipped and not processed.

[0090] Figure 5 The device for collecting time series data based on dynamic time window and fingerprint deduplication is shown, which comprises: A sliding time window creation module is used to establish a sliding time window with variable length based on the current system time, and only the incremental time series data in the window is queried, and the window slides forward with the system time; An adaptive window scaling module is used to scale the length of the sliding time window in real time based on the ratio of the amount of data collected this time to the expected amount of data after each polling, to balance the real-time performance of collection and system load; A trigger interval setting module is used to set the trigger interval of adjacent two polling to be half of the current length of the sliding time window, so that adjacent windows partially overlap in time, for capturing out-of-order arriving data; A data fingerprint generation module is used to extract key features from each time series data and generate data fingerprints; A fingerprint deduplication module is used to use a Bloom filter to preliminarily screen the data fingerprints, and then use a key-value storage system to accurately deduplicate the fingerprints that hit the preliminary screening, filter duplicate data, and ensure that the same fingerprint is collected only once.

[0091] The above device realizes the steps of the time series data collection method disclosed in the application based on dynamic time window and fingerprint deduplication when running.

[0092] The flow and block diagrams in the drawings show possible implementation ways of apparatuses, methods and computer program products according to various embodiments of the present application, including architecture, functions and operations. In these diagrams, each block can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should be noted that each block in the block diagram and / or flowchart and the combination of these blocks can use a dedicated hardware-based system to implement the specified function or operation, or a combination of dedicated hardware and computer instructions.

[0093] As Figure 6 shown, the embodiments of the present application also disclose an electronic device, comprising: a processor 310, a communication interface 320, a memory 330 for storing processor-executable computer programs, and a communication bus 340. Wherein the processor 310, the communication interface 320 and the memory 330 complete the communication among each other through the communication bus 340. The processor 310 realizes the steps of the above-mentioned time series data acquisition method based on dynamic time window and fingerprint deduplication by running the executable computer programs.

[0094] It can be understood that, in addition to including the memory and the processor, the electronic device can also contain input devices (such as a keyboard), output devices (such as a display) and other communication modules. These input devices, output devices and other communication modules all communicate with the processor through an I / O interface (i.e. input / output interface).

[0095] The operations of the present application can be implemented by using one or more program design languages or combinations thereof to write computer program codes. The program design languages include but are not limited to the following types: Object-oriented program design languages, such as Java, Smalltalk, C++, etc. Conventional procedural program design languages, such as "C" language or similar program design languages.

[0096] The execution ways of the program codes include but are not limited to: Completely on a user computer; Partially on a user computer and partially on a remote computer; As a separate software package; Completely on a remote computer or server.

[0097] In scenarios that involve a remote computer, the remote computer can be connected to the user's computer through any type of network, including, but not limited to, a local area network (LAN) or a wide area network (WAN). Still yet, the remote computer can be connected to the external computer through the Internet using an Internet service provider.

[0098] Further, the present application also discloses a computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enable the electronic device to perform each step of the time-series data collection method based on dynamic time window and fingerprint deduplication disclosed by the present application.

[0099] In the context of the present application, computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following: (1) Portable computer disk: such as floppy disk and other removable magnetic storage medium.

[0100] (2) Hard disk: including mechanical hard disk and solid state disk and other fixed storage device.

[0101] (3) Random access memory (RAM): volatile storage medium for temporarily storing data and program code.

[0102] (4) Read-only memory (ROM): non-volatile storage medium for storing fixed programs and data.

[0103] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage medium that supports multiple erasing and programming.

[0104] (6) Fiber storage device: storage medium based on fiber technology.

[0105] (7) Portable compact disc read-only memory (CD-ROM): read-only medium that stores data in the form of optical disc.

[0106] (8) Optical storage device: such as DVD, Blu-ray disc and other storage medium based on optical principle.

[0107] (9) Magnetic storage device: such as magnetic tape, magnetic disk and other storage medium based on magnetic principle.

[0108] (10) Any suitable combination of the above: for example, a combination of multiple storage media is used to meet different storage needs.

[0109] These computer-readable storage media can be used to store program code and related data described in the present application to support the running of programs and the persistent storage of data.

[0110] In particular, according to embodiments of the present application, the processes described in the flowcharts can be implemented as a computer software program. For example, embodiments of the present application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. The computer program contains program code for performing the time series data collection method based on dynamic time window and fingerprint deduplication disclosed in the present application. When the computer program is executed by a processing device, the above-mentioned functions defined in embodiments of the present application can be realized.

[0111] Although the above discussion contains a number of specific implementation details, these should not be construed as limiting the scope of the present application. The above description merely illustrates the preferred embodiment of the present application and a principle of the applied technology. It should be understood by those skilled in the art that the disclosed scope of the present application is not limited to the technical solutions formed by the specific combinations of the above technical features. Meanwhile, the present application should also cover other technical solutions formed by any combinations of the above technical features or their equivalent features without departing from the disclosed concept.

[0112] Those skilled in the art should also understand that the technical solutions recorded in the foregoing embodiments can be modified or some technical features can be replaced by equivalents without departing from the spirit and scope of the technical solutions of the embodiments of the present application. These modifications or replacements will not cause the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A time series data collection method based on dynamic time window and fingerprint deduplication, characterized in that: The method comprises: S1. Create a sliding time window of variable length based on the current system time. Query only the incremental time series data within this window, which slides forward with the system time. S2. After each poll, based on the ratio of the amount of data collected this time to the expected amount of data, the length of the sliding time window is adaptively scaled in real time to balance the real-time acquisition and system load; S3. The trigger interval between two adjacent polls is set to half the current length of the sliding time window, so that adjacent windows partially overlap in time to capture data arriving out of order; S4. Extract key features from each time series data item and generate a data fingerprint. S5. Use a Bloom filter to preliminarily screen the data fingerprints, and then use a key-value storage system to accurately deduplicate the fingerprints that hit the preliminary screening, filter out duplicate data, and ensure that the same fingerprint is only collected once.

2. The method according to claim 1, characterized in that The initial length of the sliding time window in step S1 is 60 seconds, and its length is limited to a range from a minimum of 10 seconds to a maximum of 5 minutes.

3. The method according to claim 1, characterized in that In step S2, based on the ratio of the amount of data collected this time to the expected amount of data, the length of the sliding time window is adaptively scaled in real time, wherein the rule of real-time adaptive scaling is: When the ratio of the data volume to the expected data volume is higher than 120%, the window length is reduced by 20%; When the ratio of the data volume to the expected data volume is less than 80%, the window length is increased by 20%.

4. The method according to claim 1, wherein The key features in step S4 include: Timestamp, expressed in nanosecond precision; Field values ​​are sorted by field name and concatenated. Floating-point numbers retain 6 decimal places. Tag key-value pairs are sorted in TagKey lexicographical order and then concatenated.

5. The method according to claim 1, wherein Step S4 also includes: S41. Extract key features from each time series data item; S42. Standardize and concatenate the fingerprint string according to preset rules; S43. Perform a SHA-256 hash calculation on the fingerprint string to generate a fixed-length hash value as the data fingerprint.

6. The method according to claim 1, characterized in that In step S5, the key-value storage system is Redis, and the fingerprint is stored in the form of set members, and a time to live TTL is set to control memory usage.

7. The method according to claim 6, characterized in that Step S5 also includes: S51. Perform a preliminary deduplication check on the data fingerprint using a Bloom filter. If the Bloom filter determines that the fingerprint does not exist, the data is not duplicated and is considered new for subsequent processing. S52. If the Bloom filter determines that the fingerprint may exist, it connects to Redis through Jedis to check whether the fingerprint exists in the specified Redis collection. S53. If it exists, skip the data; if it does not exist, consider the data as new data and proceed with subsequent processing.

8. A time series data acquisition device based on dynamic time window and fingerprint deduplication, characterized in that: The device comprises: The sliding time window creation module is used to establish a sliding time window of variable length based on the current system time. Only incremental time series data within the window is queried, and the window slides forward with the system time. An adaptive window scaling module is used to adaptively scale the length of the sliding time window in real time based on the ratio of the amount of data collected this time to the expected amount of data after each polling, so as to balance the real-time nature of the collection and the system load; A trigger interval setting module is used to set the trigger interval between two adjacent pollings to half the current length of the sliding time window, so that adjacent windows partially overlap in time to capture data arriving out of order; Data fingerprint generation module, used to extract key features from each time series data and generate data fingerprints; The fingerprint deduplication module is used to perform a preliminary screening of the data fingerprint using a Bloom filter, and then accurately deduplicate the fingerprints hit by the preliminary screening through a key-value storage system, filter out duplicate data, and ensure that the same fingerprint is collected only once.

9. An electronic device, characterized in that: include: memory and processor; Memory: used to store computer programs; Processor: used to execute the computer program to implement the steps of the time series data collection method based on dynamic time window and fingerprint deduplication as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program implements the steps of the time series data collection method based on dynamic time window and fingerprint deduplication according to any one of claims 1 to 7.

Images