DataX-based InfluxDB time sequence data synchronization method and device and electronic equipment

Abstract

The invention relates to an InfluxDB time sequence data synchronization method and device based on DataX. The method comprises the steps that configuration management is carried out, and InfluxDB read plug-in parameters are defined and configured; query construction and execution: segmenting the data synchronization time range into a plurality of sub-time periods, constructing a query statement, executing query, converting a result, and sending the result to a target end; incremental synchronization: recording the synchronization progress through a check point mechanism, and determining and synchronizing incremental data; and exception processing: detecting and processing connection exception, query exception and data exception. According to the method, through time range segmentation and a check point mechanism, the data synchronization efficiency is effectively improved, and the performance bottleneck and risk caused by large data volume query are avoided; incremental synchronization is supported, the data transmission quantity is reduced, and the system burden is reduced; and an exception handling mechanism guarantees the stability and accuracy of data synchronization. The method is suitable for large-scale time series data processing scenes and can meet enterprise-level data synchronization requirements.

Description

Technical Field

[0001] This application belongs to the technical field of data synchronization, and particularly relates to a method, device, computer-readable storage medium, and electronic device for synchronizing time-series data of InfluxDB based on DataX. The method of this application is applicable to scenarios of collecting and analyzing time-series data such as industrial Internet of Things, operation and maintenance monitoring, and financial market conditions. Background Art

[0002] The following problems exist in traditional time-series data synchronization solutions: (1) Performance bottleneck Limitations of InfluxDB native tools: Native tools of InfluxDB, such as influx_inspect, do not support synchronization between heterogeneous data sources. When it is necessary to interact the data in InfluxDB with other different types of data sources, this tool cannot be directly used to complete the task. In addition, in the face of large data volume export tasks, its efficiency is low, and it is difficult to meet the enterprise's demand for efficient data transmission. For example, in large-scale Internet of Things scenarios, a large amount of time series data generated by devices needs to be quickly exported for further analysis, but using influx_inspect will cause the export process to take too long, affecting the subsequent data processing process.

[0003] Storage and writing performance issues: Although the Log-Structured Merge-tree (LSM tree) structure adopted by InfluxDB is suitable for high-frequency writing, in high-concurrency writing scenarios, the writing rate may be insufficient. In addition, when the data volume continues to grow, the occupation of storage space will also become a problem. Although InfluxDB has a data compression function, when there is high-frequency writing and a large amount of historical data needs to be retained for a long time, the storage cost is still relatively high.

[0004] (2) Functional limitations Deficiencies of open-source ETL tools: Open-source ETL tools such as Logstash have many functional limitations when dealing with InfluxDB data synchronization. Its support for InfluxDB is not perfect, and it cannot make full use of the characteristics of InfluxDB for efficient data extraction, transformation, and loading. For example, Logstash may not be able to handle the complex time series data model in InfluxDB well, resulting in data loss or errors during the data synchronization process. In addition, the lack of enterprise-level features such as incremental synchronization is also an important issue. In an enterprise environment, data is constantly updated and growing, and Logstash cannot automatically identify and only synchronize newly added or changed data, which will lead to duplicate data transmission, increasing network burden and storage cost.

[0005] Query and Index Performance Issues: The query performance of InfluxDB is affected by several factors. For example, when the number of tags is too large or the discreteness of tag values is too high, the query time will increase. In addition, although the indexing mechanism of InfluxDB can quickly locate data, in a large-scale data environment, the maintenance and query optimization of indexes also face challenges. For example, although the TSI (Time Series Index) index is scalable, its performance may be affected when memory is insufficient.

[0006] (3) High development cost Requirements for processing complex logic: When developing a custom synchronization program, a large amount of complex logic needs to be processed, and data sharding is an important aspect. InfluxDB manages large-scale time series data through sharding and partitioning strategies, but in a custom synchronization program, developers need to implement the logic of data sharding themselves to ensure efficient storage and query of data. In addition, a fault recovery mechanism is also essential. During the data synchronization process, various unexpected situations such as network failures and system crashes may occur, and the custom program needs to be able to automatically detect and recover from faults to ensure the integrity and reliability of data synchronization.

[0007] High development and maintenance difficulty: Developing a custom synchronization program not only requires in-depth understanding of the architecture and data model of InfluxDB, but also requires rich programming experience and a profound understanding of the principles of data synchronization. During the development process, a large amount of debugging and optimization work needs to be carried out to ensure the performance and stability of the program. At the same time, as business requirements change and technology develops, the custom synchronization program also needs to be continuously maintained and upgraded, which further increases the development cost. Summary of the Invention

[0008] To address the above problems, this application proposes a new DataX-based InfluxDB time series data synchronization method and device, aiming to solve the problems of performance bottlenecks, functional limitations, and high development costs existing in the existing time series data synchronization methods.

[0009] This application mainly adopts the following technical strategies: 1. Development of a dedicated read plugin for InfluxDB based on the DataX framework Integrated design: Develop a read plugin specifically for the InfluxDB time series database in the DataX framework to address the problem that DataX natively does not support InfluxDB. This plugin enables efficient reading of InfluxDB data and provides basic support for time series data synchronization.

[0010] 2. Adopt an intelligent time range splitting algorithm Dynamic Sharding Mechanism: An adaptive time sharding algorithm based on the splitInterval parameter is proposed. This algorithm can automatically calculate the optimal number of shards according to the data volume, thus achieving dynamic sharding and effectively improving the efficiency of data synchronization.

[0011] Resource Protection Design: By means of sharded queries, the risk of out-of-memory (OOM) caused by a single large-volume query is avoided, thereby protecting system resources and ensuring the stability of the data synchronization process.

[0012] 3. Unified Full / Incremental Architecture Seamless Mode Switching: By setting the beginDateTime parameter, automatic switching between the full synchronization mode and the incremental synchronization mode is achieved. When the beginDateTime parameter is not set, the system defaults to the full synchronization mode; when this parameter is set, the system automatically switches to the incremental synchronization mode.

[0013] Status Sharing Mechanism: The end time of full synchronization automatically serves as the starting checkpoint for incremental synchronization, realizing status sharing between the two modes and ensuring the continuity of data synchronization.

[0014] Resource Reuse: Core components such as query construction and execution engines are fully reused in both full synchronization and incremental synchronization modes, reducing the system complexity and improving resource utilization.

[0015] 4. Distributed Incremental Synchronization Solution Checkpoint-Driven: Adopt a checkpoint mechanism in the format of a triple {lastTimestamp}|{processedCount}|{checksum}. Among them, lastTimestamp represents the latest timestamp of the last synchronization, processedCount represents the number of processed data records, and checksum is used for data integrity verification.

[0016] Dual Trigger Mechanism: Design two trigger conditions, "every 1000 records processed or per second", for saving checkpoints. This dual trigger mechanism takes into account both performance and reliability, ensuring that the synchronization status can be effectively recorded in high-throughput and low-latency scenarios.

[0017] Dynamic Adjustment Ability: The system can intelligently adjust the saving frequency of checkpoints according to the current I / O load situation. In case of sudden traffic, it can still operate stably, ensuring the continuity and accuracy of data synchronization.

[0018] Generally speaking, this application provides a method for synchronizing InfluxDB time-series data based on DataX. By extending the DataX framework and adding a new InfluxDBReader plugin, it realizes the efficient reading and synchronization of InfluxDB time-series data. This solution mainly includes the following key components: (1)Development and integration of InfluxDB read plugin Develop and integrate the InfluxDBReader plugin into the DataX framework, enabling DataX to directly read time-series data from InfluxDB. This plugin is specifically optimized for the data model and query characteristics of InfluxDB to ensure the efficiency and accuracy of data reading. Through this plugin, the DataX framework can seamlessly connect to the InfluxDB data source, providing basic support for subsequent data synchronization operations.

[0019] (2)Time range splitting and query optimization Adopt a time range splitting algorithm to implement the InfluxDB query optimization strategy. This strategy divides the query time range into multiple sub-intervals and performs query operations for each sub-interval separately, thus avoiding performance bottlenecks and system resource pressure caused by a single large-scale data query. In this way, the efficiency of data extraction can be significantly improved, supporting efficient data reading and synchronization.

[0020] (3)Incremental data synchronization mechanism Introduce a checkpoint mechanism to achieve incremental data synchronization in InfluxDB. This mechanism records the checkpoint information of the previous synchronization operation (such as timestamp, processed data volume, etc.), and only extracts the data that has been newly added or updated since the last checkpoint during subsequent synchronization. This incremental synchronization method can effectively reduce the data transmission volume, reduce the system burden, and ensure the real-time and consistency of data.

[0021] (4)Data conversion and exception handling mechanism Provide a data conversion and exception handling mechanism to ensure the accuracy of data and the stability of the synchronization process. During the data synchronization process, necessary conversion processing is performed on the data read from InfluxDB to adapt to the requirements of the target data storage or application. At the same time, a perfect exception handling mechanism is designed for possible data exceptions, network failures, system errors, etc., to ensure that problems can be detected, recorded, and processed in a timely manner when exceptions occur, guaranteeing the integrity and reliability of data synchronization.

[0022] Through the above design, this solution can efficiently and stably achieve the synchronization of InfluxDB time-series data, is applicable to large-scale time-series data processing scenarios, and meets the enterprise-level data synchronization requirements.

[0023] Specifically, the present application provides the following technical solutions: In a first aspect of the present application, a method for synchronizing InfluxDB time series data based on DataX is provided. The method includes: Configuration management: Define and configure relevant parameters in the InfluxDB read plugin; Query construction and execution: According to the configuration parameters, divide the data synchronization time range into multiple sub-time periods, construct a query statement for each sub-time period, and use the InfluxDB client to execute the query. After converting the query result into the internal format of the DataX framework, send it to the target end; Incremental synchronization: During the data synchronization process, record the synchronization progress through a checkpoint mechanism, determine the incremental data based on the last synchronization cut-off time in the checkpoint, and synchronize the incremental data; Exception handling: During the data synchronization process, detect and handle connection exceptions, query exceptions, and data exceptions respectively to ensure the stability and accuracy of data synchronization.

[0024] Further, in the method of the present application, the relevant parameters in the InfluxDB read plugin include: InfluxDB service address, username, password, database name, table name, field list for data query, filtering conditions, time splitting interval, start point of data synchronization time, and end point of data synchronization time.

[0025] Further, in the method of the present application, the method for dividing the data synchronization time range into multiple sub-time periods includes: (1) Obtain the start point of data synchronization time beginDateTime and the end point of data synchronization time endDateTime from the Configuration object. When the end point of data synchronization time is not set, the end point of data synchronization time is default to the current system time; (2) Calculate the number of segments to be divided according to the time splitting interval splitInterval. The time splitting interval splitInterval is dynamically adjusted according to the data query time span and data volume; (3) Execute the time range splitting, set a segment number for each time segment, and set the start point and end point of each time segment to the attributes beginDateTime and endDateTime respectively to form a new Configuration object; (4) Aggregate the Configuration objects of each segment to obtain a List <configuration>Set of objects.

[0026] Furthermore, in the method of the present application, the checkpoint mechanism includes the following steps: Load the checkpoint and obtain the cut-off time of the last synchronization; Compare the time of the current latest data in the InfluxDB database with the cut-off time of the last synchronization in the checkpoint to determine whether there is incremental data; If there is incremental data, perform time range splitting on the new query and construct a segmented query statement; Execute the query, and when a certain number of records are read or at regular intervals, save the last processed timestamp and the number of processed records of the current shard to the checkpoint.

[0027] Furthermore, in the method of the present application, the data in the checkpoint is saved in Redis in a compressed manner; The data format of the checkpoint is {lastTimestamp}|{processedCount}|{checksum}, where lastTimestamp is the last processed timestamp, processedCount is the number of processed records, and checksum is the checksum of the checkpoint data.

[0028] Furthermore, in the method of the present application, the exception handling includes: When a connection exception occurs, automatically retry the connection, with a maximum retry count of 6 times; When a query exception occurs, perform targeted processing for query statement spelling errors, query time exceeding the retention policy, timeout caused by excessive data volume in a single shard, lack of bucket read permission, and no data in the query range respectively; When data exception occurs, handle the data exception during the writing process by establishing constraints in the target data table or adopting a dirty data processing mechanism.

[0029] Furthermore, in the method of the present application, the query construction and execution include: constructing the query statement for each shard by looping through the set of configuration objects and using the query() method of the InfluxDB client to execute the query.

[0030] Furthermore, in the method of the present application, the InfluxDB read plugin is integrated into the DataX framework, enabling DataX to directly read time series data in InfluxDB.

[0031] The second aspect of the present application provides a DataX-based InfluxDB time series data synchronization device, which includes: Configuration management module: used to define and configure relevant parameters in the InfluxDB read plugin; Query building module: It is used to divide the data synchronization time range into multiple sub-time periods according to configuration parameters, build query statements for each sub-time period, execute queries using the InfluxDB client, and convert the query results into the internal format of the DataX framework and send them to the target end; Incremental synchronization module: It is used to record the synchronization progress through the checkpoint mechanism during data synchronization, determine the incremental data according to the last synchronization cut-off time in the checkpoint, and synchronize the incremental data; Checkpoint management module: It is used to manage the checkpoint mechanism, record the progress status of data synchronization, and ensure that the task can be resumed from the correct position after interruption; Exception handling module: It is used to detect and handle connection exceptions, query exceptions, and data exceptions respectively during data synchronization to ensure the stability and accuracy of data synchronization.

[0032] When the device runs, it implements the steps of the aforementioned InfluxDB time series data synchronization method based on DataX.

[0033] The third aspect of this application provides an electronic device, including: a memory and a processor; Memory: It is used to store computer programs; Processor: It is used to execute the computer program to implement the steps of the aforementioned InfluxDB time series data synchronization method based on DataX.

[0034] The fourth aspect of this application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the aforementioned InfluxDB time series data synchronization method based on DataX.

[0035] In summary, the InfluxDB time series data synchronization method based on DataX in this application has the following advantages: (1) Through the time range intelligent segmentation algorithm and with the help of the parallel mechanism of DataX, parallel reading can be achieved, greatly improving the reading performance of InfluxDB time series data.

[0036] (2) Incremental synchronization of InfluxDB time series data can be achieved through the checkpoint mechanism.

[0037] (3) Through the checkpoint mechanism, automatic task recovery after program failure can also be achieved.

[0038] Other features and advantages of the present application will be elaborated in detail in the subsequent specification, or can be understood by implementing the relevant technical solutions of the present application. The objectives and other advantages of the present application can be achieved by the technical features and means clearly pointed out in the specification, claims, and drawings, and obtained through the implementation process of these technical contents. Brief Description of the Drawings

[0039] In order to more clearly elaborate the technical solutions of the embodiments of the present application, the drawings involved in the description of the embodiments will be briefly introduced below. It should be noted that the drawings only show some embodiments of the present application. For those skilled in the art, other relevant drawings can be derived based on these drawings without creative efforts.

[0040] Figure 1 It is the overall implementation flowchart of the InfluxDB time series data synchronization method based on DataX of the present application.

[0041] Figure 2 It is the operation flowchart of the query construction module in the device of the present application.

[0042] Figure 3 It is the operation flowchart of the incremental synchronization module in the device of the present application.

[0043] Figure 4 It is the composition structure diagram of the InfluxDB time series data synchronization device based on DataX of the present application.

[0044] Figure 5 It is the structural schematic diagram of the electronic device provided by the embodiment of the present application. Detailed Description of the Embodiments

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be clear that the described embodiments are only some embodiments of the present application, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the protection scope of the present application.

[0046] In this document, the term "including" and any of its variants (such as "including having", "including") are open-ended expressions and should be understood as "including but not limited to", that is, the listed content is not an exhaustive list and may also include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on", that is, the referred basis or condition may not be the only factor and may also involve other relevant factors. The term "an embodiment" should be understood as "at least one embodiment", that is, the described embodiment is not the only possible implementation and there may be other similar embodiments.

[0047] In this application, when the terms "a" and "plural" are used to modify relevant elements or features, their expressions are illustrative rather than restrictive. Unless otherwise clearly stated in the context, "a" should be understood as "at least one", and "plural" should be understood as "at least two". Those skilled in the art should make a reasonable interpretation of these terms according to the semantic and logical relationships in the context to ensure that they cover the possibility of "one or more".

[0048] Figure 1 The following shows the overall implementation process of the InfluxDB time series data synchronization method based on DataX provided by this application, including the following steps: Configuration management: Define and configure relevant parameters in the InfluxDB read plugin, including: InfluxDB service address, username, password, database name, table name, field list for data query, filtering conditions, time segmentation interval, start point of data synchronization time, and end point of data synchronization time; Query construction and execution: According to the configuration parameters, divide the data synchronization time range into multiple sub-time periods, construct query statements for each sub-time period, and use the InfluxDB client to execute the query. After converting the query results into the internal format of the DataX framework, send them to the target end; Incremental synchronization: During the data synchronization process, record the synchronization progress through the checkpoint mechanism, determine the incremental data according to the last synchronization end time in the checkpoint, and synchronize the incremental data; Exception handling: During the data synchronization process, detect and handle connection exceptions, query exceptions, and data exceptions respectively to ensure the stability and accuracy of data synchronization.

[0049] To more clearly elaborate on the technical solution of this application, the following will be further described through embodiments in specific scenarios.

[0050] In this application, by developing and integrating the InfluxDB read plugin into the DataX framework, the extraction of time series data from the InfluxDB time series database is achieved. The InfluxDB time series data synchronization device based on DataX provided in this application includes the following functional modules: (1) Configuration management module The configuration management module is one of the core components of this solution and is responsible for defining and configuring various parameters in the InfluxDB read plugin to ensure that the data synchronization process can be executed according to the predefined rules and conditions.

[0051] The configuration management module needs to configure the following key parameters: endpoint: InfluxDB service address.

[0052] username: Username.

[0053] password: Password.

[0054] database: Database name.

[0055] measurement: Table name.

[0056] fields: List of fields for data query.

[0057] where: Filter condition.

[0058] splitInterval: Time segmentation interval, such as 1 day, 1 hour.

[0059] beginDateTime: Starting point of data synchronization time.

[0060] endDateTime: Ending point of data synchronization time.

[0061] (2) Query construction module The query construction module is the core component for integrating DataX with the InfluxDB database and is responsible for reading data from the InfluxDB database.

[0062] As Figure 2 shown, the main functions of the query construction module include: Time range segmentation: Segment the large time range into multiple small time periods for parallel processing and improved synchronization efficiency.

[0063] Construct InfluxQL (Influx Query Language) for query: Based on the segmented time periods, construct the InfluxQL for querying the InfluxDB database.

[0064] Execute query: Use the InfluxDB client to execute InfluxQL queries, and after data transformation, send the query results to the target end through the DataX framework.

[0065] 1. Time range splitting In this application, a scheme of splitting the query of the source InfluxDB database according to the time range is adopted. This scheme is based on the following design: As a time series database, InfluxDB will accumulate a large amount of data over time. When synchronizing data for a long time range (such as several months or years), a single query will return an extremely large result set, leading to problems such as the risk of out-of-memory (OOM), query timeouts, and high network transmission pressure.

[0066] The TSM storage engine used by InfluxDB organizes data by time range. Using sharded queries can improve the locality access efficiency of the storage engine.

[0067] The time range splitting algorithm adopted in this application is as follows: (1) Obtain the start point beginDateTime of the data synchronization time and the end point endDateTime of the data synchronization time from the Configuration object. When the end point of the data synchronization time is not set, the end point of the data synchronization time is default to the current system time; (2) Calculate the number of segments to be split (rounded up) according to the time splitting interval splitInterval. The time splitting interval splitInterval can be dynamically adjusted according to the data query time span and data volume; (3) Execute time range splitting, set the segment number for each time segment, and set the start point and end point of each time segment to the attributes beginDateTime and endDateTime respectively to form a new Configuration object; (4) Aggregate the Configuration objects of each segment to obtain a List <configuration>Set of objects.

[0068] 2. Construct the query InfluxQL The query language of InfluxDB is InfluxQL, whose syntax is similar to SQL but optimized for time series data.

[0069] Its basic query syntax is: SELECT <field_keys>[, <tag_keys>|*] FROM <measurement_name> [WHERE <conditions> [GROUP BY<tags|time>] [ORDER BY time [ASC|DESC]] [LIMIT <n>[OFFSET <n> Description: (1)SELECT <field_keys>[,<tag_keys>|*] Function: Specify the fields or tags to be returned. field_keys: Required, columns that store actual values (such as temperature, CPU usage); tag_keys: Optional, index columns (usually string type), used for efficient filtering and grouping.

[0070] (2)FROM <measurement_name> Function: Specify the table to be queried (referred to as measurement in InfluxDB).

[0071] (3)WHERE <conditions> Function: Filter data, support conditional filtering for tags, fields, and time.

[0072] (4)GROUP BY<tags|time> Function: Group by tags or time, often used in conjunction with aggregation functions.

[0073] (5)ORDER BY time [ASC|DESC] Function: Sort by timestamp (InfluxQL only supports sorting by time).

[0074] (6)LIMIT <n>[OFFSET <m> Function: Paging to control the size of the result set.

[0075] Table 1 InfluxDB Example Table

[0076] In this step, the List obtained in the previous step will be looped through. <configuration>A set of objects is used to loop through and construct the query statements for each shard. The query statement for a single shard constructed according to the above scheme is as follows: SELECT device_id, temperature FROM iot_sensors WHERE time>= '2022-02-15T00:00:00Z' AND time<'2022-02-16T23:59:59Z' 3. Execute the query Use the query() method of the InfluxDB client to execute the InfluxQL query. After obtaining the results, data conversion is required. Each field in the InfluxDB query results is converted into the internal format of the DataX framework and finally encapsulated into a Record object, and then the data is sent to the Writer plugin through RecordSender.

[0077] (III) Incremental Synchronization Module The incremental synchronization module is the core module for implementing incremental reading of InfluxDB time series data.

[0078] As Figure 3 shown, the incremental synchronization process is as follows: (1) Load the checkpoint First, load the checkpoint to obtain the cut-off time of the last synchronization saved in the checkpoint.

[0079] (2) Data comparison Obtain the time of the current latest data in the InfluxDB database and compare it with the cut-off time of the last synchronization in the obtained checkpoint. If the above two time values are not equal, it means that incremental data has been generated and incremental data synchronization needs to continue.

[0080] If the above two time values are equal, it means that no incremental data has been generated, and data synchronization will no longer continue.

[0081] (3) Time range splitting Use the cut-off time of the last synchronization in the obtained checkpoint as the start point beginDateTime of the new query data synchronization time, and then use the same time range splitting method as in the query construction module to split the time range of the new query.

[0082] (4) Construct segmented query statements Use the same method as in the query construction module to construct the InfluxQL query statement.

[0083] (5) Execute the query Execute the query using the same method as in the query construction module.

[0084] (6)Save checkpoint During the incremental data synchronization process, when every 1000 records are read or every 1 second has elapsed, the timestamp of the last processed record, the number of processed records, etc. in the current shard will be saved to the checkpoint.

[0085] (IV)Checkpoint management module The checkpoint mechanism is the core of incremental synchronization. It records the progress status of data synchronization and ensures that the task can resume from the correct position after interruption.

[0086] (1)Checkpoint saving method In order to enable incremental synchronization in a distributed environment, this solution adopts the method of caching checkpoint data in Redis (Remote Dictionary Server, an open-source, memory-based key-value storage system that supports multiple data structures and can be designed for caching systems).

[0087] (2)Checkpoint data format In this solution, the checkpoint data format is defined as: {lastTimestamp}|{processedCount}|{checksum} Among them, lastTimestamp is the timestamp of the last processed record (in ISO8601 format), processedCount is the number of processed records, and checksum is the checksum of the checkpoint data (MD5 value), separated by the vertical bar |.

[0088] (3)Trigger timing for saving checkpoint When reading data, it is designed to trigger a checkpoint save every time 1000 records are read or every 1 second has elapsed.

[0089] When using the interval time for checkpoint saving, the optimal time interval for checkpoint saving can also be dynamically calculated based on the system I / O load and network status.

[0090] When the read operation ends normally, the last checkpoint save will also be triggered.

[0091] Among them, using the task id as the key and the checkpoint data as the value, it is saved to Redis. In order to save Redis memory space, the checkpoint data will be compressed in this solution.

[0092] (4)Loading of checkpoint When the task starts, first, using the task ID as the key, retrieve the saved checkpoint data from Redis.

[0093] If no checkpoint data is retrieved, it indicates a new data synchronization task, and data synchronization will directly start from the set starting point of the data synchronization time.

[0094] If checkpoint data is retrieved, it is necessary to verify the timestamp format, the number of records, and the value of the checksum in the checkpoint. After passing the verification, data synchronization will start from the last processed time saved in the checkpoint as the starting point of the data synchronization time.

[0095] (V) Exception Handling Module The exception handling module is the core module to ensure the stability of the data synchronization job and the accuracy of the data. In this solution, the following three types of exception situations are handled respectively: connection exception, query exception, and data exception.

[0096] (1) Connection Exception When connecting to the InfluxDB server through the client and situations such as connection timeout or authentication failure occur, the program will automatically retry the connection, with a maximum of 6 retry attempts.

[0097] Table 2 Connection Exception Example Table

[0098] (2) Query Exception In this solution, for exception scenarios such as syntax spelling errors in InfluxDB query statements, query time exceeding the retention policy, and timeout caused by excessive data volume in a single shard, targeted handling is carried out respectively.

[0099] Table 3 Query Exception Example Table

[0100] (3) Data Exception During the data synchronization process, for data exception scenarios, data exceptions during the writing process will be handled by establishing constraints in the target data table or by means of a dirty data handling mechanism.

[0101] After the data synchronization is completed, means such as comparison and verification are used to ensure the accuracy of the data.

[0102] Figure 4 The following shows a DataX-based InfluxDB time series data synchronization device proposed in this application, including: Configuration Management Module: Used to define and configure relevant parameters in the InfluxDB read plugin; Query building module: used to divide the data synchronization time range into multiple sub - time periods according to configuration parameters, build query statements for each sub - time period, execute the queries using the InfluxDB client, and convert the query results into the internal format of the DataX framework and send them to the target end; Incremental synchronization module: used to record the synchronization progress through the checkpoint mechanism during data synchronization, determine the incremental data according to the last synchronization cut - off time in the checkpoint, and synchronize the incremental data; Checkpoint management module: used to manage the checkpoint mechanism, record the progress status of data synchronization, and ensure that it can resume from the correct position after the task is interrupted; Exception handling module: used to detect and handle connection exceptions, query exceptions, and data exceptions respectively during data synchronization to ensure the stability and accuracy of data synchronization.

[0103] When the above - mentioned device runs, it implements the steps of the InfluxDB time - series data synchronization method based on DataX disclosed in this application.

[0104] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementations of devices, methods, and computer program products according to various embodiments of this application, including system architectures, functions, and operations. In these figures, each block may represent a module, a program segment, or a part of code, which contains one or more executable instructions for implementing the specified logical function. It should be noted that each block in the block diagram and / or flowchart, as well as combinations of these blocks, can be implemented using a dedicated hardware - based system to implement the specified function or operation, or can be implemented through a combination of dedicated hardware and computer instructions.

[0105] As Figure 5 shown, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing computer programs executable by the processor, and a communication bus 340. Among them, the processor 310, the communication interface 320, and the memory 330 complete mutual communication through the communication bus 340. The processor 310 runs the executable computer program to implement the steps of the above - mentioned InfluxDB time - series data synchronization method based on DataX.

[0106] It can be understood that in addition to including a memory and a processor, this electronic device may also include an input device (such as a keyboard), an output device (such as a display), and other communication modules. These input devices, output devices, and other communication modules communicate with the processor through the I / O interface (i.e., the input / output interface).

[0107] The operations of this application can be implemented by writing computer program code using one or more programming languages or combinations thereof. The programming languages include, but are not limited to, the following types: Object-oriented programming languages, such as Java, Smalltalk, C++, etc.; Conventional procedural programming languages, such as the "C" language or similar programming languages.

[0108] The execution modes of the program code include, but are not limited to: Executing entirely on the user's computer; Executing partially on the user's computer and partially on a remote computer; Executing as an independent software package; Executing entirely on a remote computer or server.

[0109] In scenarios involving a remote computer, the remote computer can be connected to the user's computer through any type of network connection, and the network includes, but is not limited to, a local area network (LAN) or a wide area network (WAN). In addition, the remote computer can also be connected to an external computer through an Internet service provider, such as by using the Internet for connection.

[0110] Furthermore, this application also discloses a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by the processor of an electronic device, the electronic device can execute each step of the InfluxDB time series data synchronization method based on DataX disclosed in this application.

[0111] In the context of this application, a computer-readable storage medium refers to a tangible medium that can store computer program code and related data. Specific examples include, but are not limited to, the following: (1) Portable computer disks: Removable magnetic storage media such as floppy disks.

[0112] (2) Hard disks: Fixed storage devices including mechanical hard disks and solid-state drives.

[0113] (3) Random access memory (RAM): A volatile storage medium for temporarily storing data and program code.

[0114] (4) Read-only memory (ROM): A non-volatile storage medium for storing fixed programs and data.

[0115] (5) Erasable programmable read-only memory (EPROM) or flash memory: A non-volatile storage medium that supports multiple erasures and programming.

[0116] (6) Optical fiber storage device: A storage medium based on optical fiber technology.

[0117] (7) Portable compact disc read-only memory (CD-ROM): A read-only medium for storing data in the form of optical discs.

[0118] (8) Optical storage device: Storage media based on optical principles such as DVDs, Blu-ray discs, etc.

[0119] (9) Magnetic storage device: Storage media based on magnetic principles such as magnetic tapes, magnetic disks, etc.

[0120] (10) Any suitable combination of the above: For example, combining multiple storage media to meet different storage requirements.

[0121] These computer-readable storage media can be used to store the program code and related data described in this application to support the operation of the program and the persistent storage of data.

[0122] In particular, according to the embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, the embodiments of this application relate to a computer program product that includes a computer program carried on a non-transitory computer-readable medium. This computer program contains program code for executing the InfluxDB time series data synchronization method based on DataX disclosed in this application. When this computer program is executed by a processing device, it can implement the above functions defined in the embodiments of this application.

[0123] Although the above discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is only a preferred embodiment of this application and an explanation of the applied technical principles. Those skilled in the art should understand that the disclosed scope of this application is not limited to the technical solutions formed by the specific combination of the above technical features. At the same time, this application should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept.

[0124] Those skilled in the art should also understand that they can modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.< / configuration> ​< / m> < / n> < / conditions> ​< / n> < / n> ​< / conditions> < / configuration> < / configuration>

Claims

1. A method for synchronizing InfluxDB time series data based on DataX, characterized in that The method includes: Configuration management: Defining and configuring relevant parameters in the InfluxDB read plugin; Query construction and execution: According to the configuration parameters, splitting the data synchronization time range into multiple sub-time periods, constructing query statements for each sub-time period, and using the InfluxDB client to execute the queries. After converting the query results into the internal format of the DataX framework, send them to the target end; Incremental synchronization: During the data synchronization process, recording the synchronization progress through the checkpoint mechanism, determining the incremental data based on the last synchronization cut-off time in the checkpoint, and synchronizing the incremental data; Exception handling: During the data synchronization process, detecting and handling connection exceptions, query exceptions, and data exceptions respectively to ensure the stability and accuracy of data synchronization.

2. The method according to claim 1, wherein The relevant parameters in the InfluxDB read plugin include: InfluxDB service address, username, password, database name, table name, field list for data query, filtering conditions, time splitting interval, start point of data synchronization time, and end point of data synchronization time.

3. The method according to claim 1, wherein The time range splitting method for splitting the data synchronization time range into multiple sub-time periods includes: (1) Obtaining the start point of data synchronization time beginDateTime and the end point of data synchronization time endDateTime from the Configuration object. When the end point of data synchronization time is not set, the end point of data synchronization time is default to the current system time; (2) Calculating the number of segments to be split according to the time splitting interval splitInterval, and the time splitting interval splitInterval is dynamically adjusted according to the data query time span and data volume; (3) Performing time range splitting, setting the segment number for each time segment, and setting the start point and end point of each time segment to the attributes beginDateTime and endDateTime respectively to form a new Configuration object; (4) Aggregate the Configuration objects of each segment to obtain a List <configuration>Object collection. < / configuration> 4. The method according to claim 1, wherein The checkpoint mechanism includes the following steps: Loading the checkpoint and obtaining the cut-off time of the last synchronization; Comparing the time of the current latest data in the InfluxDB database with the cut-off time of the last synchronization in the checkpoint to determine whether there is incremental data; If there is incremental data, perform time range splitting on the new query and construct segmented query statements; Execute the query, and save the last processed timestamp and the number of processed records of the current shard to the checkpoint every time a certain number of records are read or at a certain time interval.

5. The method according to claim 4, characterized in that The data in the checkpoint is saved in Redis in a compressed manner; The data format of the checkpoint is {lastTimestamp}|{processedCount}|{checksum}, where lastTimestamp is the last processed timestamp, processedCount is the number of processed records, and checksum is the checksum of the checkpoint data.

6. The method according to claim 1, wherein The exception handling includes: When a connection exception occurs, automatic connection retry is performed, and the maximum number of retries is 6 times; When a query exception occurs, targeted processing is performed respectively for query statement spelling errors, query time exceeding the retention policy, timeout caused by excessive data volume in a single shard, lack of bucket read permission, and no data in the query range; When data exception occurs, data exceptions during the writing process are handled by establishing constraints in the target data table or adopting a dirty data processing mechanism.

7. The method according to claim 1, characterized in that The query construction and execution include: by looping through the configuration object set, constructing the query statement for each shard, and using the query() method of the InfluxDB client to execute the query.

8. The method according to claim 1, characterized in that The InfluxDB read plugin is integrated into the DataX framework, enabling DataX to directly read time series data in InfluxDB.

9. An InfluxDB time series data synchronization device based on DataX, characterized in that, The device includes: Configuration management module: used to define and configure relevant parameters in the InfluxDB read plugin; Query construction module: used to divide the data synchronization time range into multiple sub-time periods according to the configuration parameters, construct query statements for each sub-time period, execute the query using the InfluxDB client, and convert the query results into the internal format of the DataX framework and send them to the target end; Incremental synchronization module: used to record the synchronization progress through the checkpoint mechanism during the data synchronization process, determine the incremental data according to the last synchronization cut-off time in the checkpoint, and synchronize the incremental data; Checkpoint management module: used to manage the checkpoint mechanism, record the progress status of data synchronization, and ensure that it can be resumed from the correct position after the task is interrupted; Exception handling module: used to detect and handle connection exceptions, query exceptions, and data exceptions respectively during the data synchronization process to ensure the stability and accuracy of data synchronization.

10. An electronic device, characterized in that, Includes: Memory and processor; Memory: used to store computer programs; Processor: used to execute the computer program to implement the steps of the InfluxDB time series data synchronization method based on DataX as described in any one of claims 1-8.

Images