Method and device for guaranteeing consistency of distributed data in data acquisition

Abstract

The invention discloses a method and device for guaranteeing the consistency of distributed data in data acquisition, and the method comprises the steps: taking a SeaCraabSink interface as a core entrance of a framework, defining a creation method of a target data writer, a target data submitter and a target data aggregation submitter, and being responsible for the full-life-cycle management of a Sink component; the target data writer interface is used for bearing core logic of data writing and is used for writing a single data element, preparing submitted information and periodically storing state snapshots; the target data submitter interface is responsible for a submission process and is provided with a data submission canceling method so as to cope with a rollback requirement when submission fails; the target data aggregation submitter interface serves as a high-level version of the target data aggregation submitter interface, a data integration method is introduced, and multiple pieces of submission information are integrated into single aggregation information to be submitted. According to the method and the device, the accuracy and the integrity of data submission can be ensured in a distributed environment for a highly uncertain fault scene.

Description

Technical Field

[0001] The present invention relates to the field of distributed data collection, and in particular to a method and device for ensuring the consistency of distributed data collected. Background Art

[0002] In a distributed system architecture, maintaining data consistency is a key element to ensure reliable operation of the system, and is directly related to user confidence in the system. The data consistency mechanism ensures that the system can maintain the accuracy and integrity of data between nodes even under extreme conditions such as network interruption or node failure. However, achieving this goal faces multiple challenges such as transaction processing, state consistency, and partition fault tolerance, especially in a distributed environment where the complexity increases significantly. Summary of the invention

[0003] In order to enhance the consistency in the distributed data collection process, the present invention provides a method and device for ensuring the consistency of distributed data collection, aiming to ensure the accuracy and completeness of data submission in a distributed environment even in the face of highly uncertain failure scenarios, and is particularly suitable for large-scale data processing systems that require extremely high reliability and strong consistency guarantees.

[0004] To achieve the above purpose, the present invention adopts the following technical solutions:

[0005] In one embodiment of the present invention, a method for ensuring the consistency of distributed data collection is proposed, the method comprising:

[0006] SeaCrabSink interface, as the core entry of SeaCrabSink framework, defines the creation methods of target data writer, target data submitter and target data aggregation submitter, and is responsible for the full life cycle management of Sink component, including initialization, component construction and serialization processing;

[0007] The target data writer interface carries the core logic of data writing and has methods for writing, pre-committing, and snapshot status management. It is used for writing a single data element, preparing submission information, and periodically saving status snapshots.

[0008] The target data submitter interface is responsible for the submission process. Its data submission method receives the submission information and attempts to pass it to the external data receiving end. It is also equipped with a data submission cancellation method to cope with the rollback requirements when the submission fails.

[0009] The target data aggregation submitter interface, as an advanced version of the target data submitter interface, introduces a data integration method to integrate multiple submission information into a single aggregation information and then submit it.

[0010] Further, the pre-commit method of the target data writer interface constitutes the first phase of the two-phase commit, and the data submission method of the target data committer interface or the target data aggregation committer interface constitutes the second phase of the two-phase commit;

[0011] Phase 1: The coordinator node sends a commit preparation request to all participant nodes and waits for their replies. After receiving the commit preparation request, each participant node executes data updates related to the distributed transaction and saves the operation results in the local log. If the participant node executes successfully, it returns a "complete" message to the coordinator node. When the coordinator node receives the return message from all participant nodes, the entire distributed transaction enters the second phase. If one participant node returns a failure, the coordinator node sends a rollback transaction request to all participant nodes, indicating that the distributed transaction execution has failed.

[0012] Phase 2: The coordinator node decides whether to commit the transaction based on the responses from the participant nodes. If the coordinator node receives all responses agreeing to commit, it sends a commit request to all participant nodes and waits for their confirmation. After receiving the commit request, the participant node updates the operation result to the database and sends a confirmation message to the coordinator node. After receiving the confirmation messages from all participant nodes, the coordinator node finally decides whether to commit or roll back the transaction and notifies all participant nodes of the decision.

[0013] Furthermore, the data submission methods of the target data submitter and the target data aggregation submitter interfaces are both idempotent.

[0014] In one embodiment of the present invention, a device for ensuring the consistency of distributed data collection is also proposed, the device comprising:

[0015] SeaCrabSink interface definition module, which defines the creation methods of target data writer, target data submitter and target data aggregation submitter, and is responsible for the full life cycle management of Sink components, including initialization, component construction and serialization processing;

[0016] The target data writer interface module carries the core logic of data writing and has writing, pre-commitment and snapshot status management methods, which are used for writing a single data element, preparing submission information and periodically saving status snapshots;

[0017] The target data submitter interface module is responsible for the submission process. Its data submission method receives the submission information and attempts to pass it to the external data receiving end. It is also equipped with a data submission cancellation method to cope with the rollback requirements when the submission fails.

[0018] The target data aggregation submitter interface module, as an advanced version of the target data submitter interface, introduces a data integration method to integrate multiple submission information into a single aggregation information and then submit it.

[0019] Further, the pre-commit method of the target data writer interface constitutes the first phase of the two-phase commit, and the data submission method of the target data committer interface or the target data aggregation committer interface constitutes the second phase of the two-phase commit;

[0020] Phase 1: The coordinator node sends a commit preparation request to all participant nodes and waits for their replies. After receiving the commit preparation request, each participant node executes data updates related to the distributed transaction and saves the operation results in the local log. If the participant node executes successfully, it returns a "complete" message to the coordinator node. When the coordinator node receives the return message from all participant nodes, the entire distributed transaction enters the second phase. If one participant node returns a failure, the coordinator node sends a rollback transaction request to all participant nodes, indicating that the distributed transaction execution has failed.

[0021] Phase 2: The coordinator node decides whether to commit the transaction based on the responses from the participant nodes. If the coordinator node receives all responses agreeing to commit, it sends a commit request to all participant nodes and waits for their confirmation. After receiving the commit request, the participant node updates the operation result to the database and sends a confirmation message to the coordinator node. After receiving the confirmation messages from all participant nodes, the coordinator node finally decides whether to commit or roll back the transaction and notifies all participant nodes of the decision.

[0022] Furthermore, the data submission methods of the target data submitter and the target data aggregation submitter interfaces are both idempotent.

[0023] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aforementioned method for ensuring the consistency of distributed data collected is designed.

[0024] In one embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program for executing the method for ensuring the consistency of distributed data in data collection.

[0025] Beneficial effects:

[0026] 1. The present invention ensures the exact semantics of data. These components help reduce the risk of data duplication or loss and ensure the consistency and accuracy of data.

[0027] 2. The present invention improves the speed and efficiency of data processing by optimizing the process of data writing and submission, especially for large-scale data stream processing tasks.

[0028] 3. The present invention simplifies the data management process and reduces development and maintenance costs through integrated management and submission strategies.

[0029] 4. The present invention supports transactional commit or rollback operations. These components enhance the robustness of the system and maintain data consistency even when a failure occurs.

[0030] 5. The present invention optimizes specific databases (such as ClickHouse) to meet the needs of high-performance data processing, especially when fast query response is required. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flow chart of a method for ensuring consistency of distributed data collected by the present invention;

[0032] Figure 2 This is a schematic diagram of the first phase interaction of the two-phase submission of the present invention;

[0033] Figure 3 It is a schematic diagram of the second phase interaction of the two-phase submission of the present invention;

[0034] Figure 4 This is a flow chart of a method for ensuring consistency of distributed data collected according to an embodiment of the present invention;

[0035] Figure 5 It is a schematic diagram of the structure of a device for ensuring the consistency of distributed data collected by the present invention;

[0036] Figure 6 It is a schematic diagram of the computer device structure of the present invention. DETAILED DESCRIPTION

[0037] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and design the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0038] Those skilled in the art will appreciate that the embodiments of the present invention may be designed as a device, apparatus, equipment, method or computer program product. Therefore, the present disclosure may be specifically designed in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0039] According to an implementation of the present invention, a method for ensuring the consistency of distributed data in data collection is proposed. The SeaCrabSink interface, as the core entry of the SeaCrabSink framework, defines the creation methods of the target data writer, the target data submitter and the target data aggregation submitter, and is responsible for the full life cycle management of the Sink component, including initialization, component construction and serialization processing; the target data writer interface, which carries the core logic of data writing, has writing, pre-submission and snapshot state management methods, which are used for writing a single data element, preparing submission information and periodically saving state snapshots; the target data submitter interface, which is responsible for the submission process, and its data submission method receives the submission information and tries to pass it to the external data receiving end, and is equipped with a data submission cancellation method to cope with the rollback requirement when the submission fails; the target data aggregation submitter interface, as an advanced version of the target data submitter interface, introduces a data integration method to integrate multiple submission information into a single aggregation information and then submit it.

[0040] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.

[0041] Figure 1 Schematic diagram of the method for ensuring the consistency of distributed data collection in the present invention. Figure 1 As shown in the figure, the design of these components fully considers the reliability, consistency and performance of data processing, providing a solid foundation for building a robust big data processing system. Through the application of these components, a series of challenges in data processing in distributed systems can be effectively solved, including data consistency, fault tolerance and efficient batch processing. The details are as follows:

[0042] (I) SeaCrabSink framework design

[0043] 1. SeaCrabSink interface (target data writer unified interface): As the core entry of the SeaCrabSink framework, it defines the creation methods of SinkWriter (target data writer), SinkCommitter (target data committer) and SinkAggregatedCommitter (target data aggregate committer), and is responsible for the full life cycle management of Sink components, including initialization, component construction and serialization processing.

[0044] 2. SinkWriter interface: carries the core logic of data writing, with write, prepareCommit and snapshotState methods, which are used for writing a single data element, preparing submission information and saving state snapshots. The latter can be used for periodic fault recovery.

[0045] 3. SinkCommitter interface: responsible for the submission process. Its commit (data submission) method receives the submission information and attempts to pass it to the external data receiving end. It is also equipped with a data submission cancellation method abort() to cope with the rollback requirements when the submission fails.

[0046] 4. SinkAggregatedCommitter interface: As an advanced version of the SinkCommitter interface, it introduces the combine method to integrate multiple submission information into a single aggregated information and then submit it, aiming to improve the consistency of data processing in a distributed environment.

[0047] 2. Core Strategies and Mechanisms

[0048] The consistency and integrity of data during submission is achieved through multiple layers of mechanisms.

[0049] Here are the main strategies and mechanics:

[0050] 1. Two-Phase Commit:

[0051] The prepareCommit method of the SinkWriter interface constitutes the first phase of the two-phase commit. In this phase, the system prepares the data to be submitted, but has not actually submitted it. The commit method of the SinkCommitter interface or the SinkAggregatedCommitter interface constitutes the second phase, where the commit operation is actually performed. This two-phase approach allows the system to ensure that all necessary preparations have been completed before the actual submission, increasing the reliability of the submission process. Two-phase commit usually includes two main phases:

[0052] Phase 1 (preparation phase):

[0053] like Figure 2 As shown in the figure, the coordinator node sends a commit preparation request to all participant nodes and waits for their replies. After receiving the commit preparation request, each participant node will perform data updates related to the distributed transaction and save the operation results in the local log. If the participant node executes successfully, the transaction will not be committed for the time being, but a "completed" message will be returned to the coordinator node. When the coordinator node receives the return message from all participant nodes, the entire distributed transaction will enter the second stage. If a participant node returns a failure in the first stage, the coordinator node will send a request to roll back the transaction to all participant nodes, that is, the distributed transaction execution fails.

[0054] Phase 2 (Submission Phase):

[0055] like Figure 3 As shown in the figure, the coordinator node decides whether to commit the transaction based on the responses of the participant nodes. If the coordinator node receives all the responses that agree to commit, it will send a data submission request to all participant nodes and wait for the confirmation of the participant nodes. After receiving the data submission request, the participant node updates the operation result to the database and sends a confirmation message to the coordinator node. After receiving the confirmation messages from all participant nodes, the coordinator node finally decides to commit or roll back the transaction and notifies all participant nodes of the decision.

[0056] 2. Idempotent submission:

[0057] The commit method of the SinkCommitter interface and the SinkAggregatedCommitter interface is designed to be idempotent. This means that even if the same commit operation is executed multiple times (for example, due to retries), it will not cause data duplication or inconsistency, and the effect is the same as if it was executed only once. The idempotent commit mechanism ensures data consistency in the face of network problems or system crashes.

[0058] 3. Status snapshot:

[0059] The snapshotState method of the SinkWriter interface allows to periodically save the state of the target data writer. These states can be used to recover the operation in case of failure.

[0060] By saving state periodically, the system can recover from the most recent consistent point after a failure, rather than starting from scratch.

[0061] 4. Rollback mechanism:

[0062] Both the SinkCommitter interface and the SinkAggregatedCommitter interface provide a data submission cancellation method. In the event of a submission failure, this method can be used to roll back some of the operations that have been executed. This ensures that the system can return to a consistent state when an error occurs during the submission process.

[0063] 5. Aggregate submission:

[0064] The combine method of SinkAggregatedCommitter allows multiple commit messages to be merged into one. This is particularly useful in a distributed environment to reduce network communication and improve overall consistency. Through aggregation, the system can process multiple write operations in one transaction, thereby enhancing data consistency.

[0065] 6.Serialization support:

[0066] The SeaCrabSink interface provides methods to obtain various types of serializers. Correct serialization ensures the integrity of data when transmitting in a distributed environment.

[0067] 7. Recoverable Writer:

[0068] The restoreWriter method of the SeaCrabSink interface (target data writing interruption, resume writing method) allows to restore the writer from a previously saved state. This is very useful when restarting a task after a failure to ensure data continuity and integrity.

[0069] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0070] In order to explain the above method for ensuring the consistency of distributed data in data collection more clearly, a specific embodiment is used for illustration below. However, it should be noted that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation to the present invention.

[0071] Example:

[0072] like Figure 4 As shown in the figure, for different application scenarios, we have designed aggregation committers based on the file system, ClickHouse file-level aggregation committers, JDBC-based aggregation committers, and JDBC multi-table aggregation committers. They aim to ensure reliable transmission and persistent storage of data in a distributed environment, while supporting transactional operations and maintaining data consistency and integrity.

[0073] 1. File system file submitter

[0074] FileSinkAggregatedCommitter is specifically used to handle file submission and management operations on the file system. Its design goal is to ensure that data is reliably written to the file system in large-scale data processing jobs and support transactional commit or rollback operations. The main implementation includes:

[0075] (1) Initialize the file system proxy: Create an instance of FileSystemProxy (file system proxy) to perform various file system operations, such as moving and deleting files.

[0076] (2) Configuration file sink: Create a FileSinkConfig configuration object to store configuration information about the file sink, such as the path of the recorded file, whether to open the record file flag, etc. Initialize the FileSystemProxy by passing in FileConf (file system configuration information) to interact with the file system. In addition, an additional FileSinkConfig parameter is received to customize the behavior according to the specific configuration.

[0077] (3) Commit operation: The commit method receives a list of multiple FileAggregatedCommitInfo (file aggregate commit information), which represents a set of file operations that need to be committed (i.e. finalized). It traverses each FileAggregatedCommitInfo, renames the temporary files in each transaction directory (from temporary names to formal names), and then deletes the transaction directory. If any step fails, the exception will be caught and the erroneous commit information will be recorded and returned for subsequent processing or retry.

[0078] (4) Merge operation: The combine method is used to merge multiple simple commit messages into an aggregate commit message. It merges the file move mappings in all the input commit messages. If the configuration allows, it also records all the written file names into a specified log file for auditing or fault recovery.

[0079] (5) Rollback operation: When the commit method fails, call the abort (cancel data submission) method to perform a rollback operation. Try to restore any previous modifications to maintain system consistency. The entire execution process is to traverse the provided aggregate commit information list, try to undo all previous rename operations (if the new file exists and the old file no longer exists, change the file back to its original name), and clean up the transaction directory. If an exception occurs during the operation, log it.

[0080] (6) Release resources: The close method is used to release resources and close the connection with the file system.

[0081] 2. ClickHouse File Submitter

[0082] ClickhouseFileSinkAggCommitter (Clickhouse file write aggregation committer) is designed for ClickHouse database and aims to batch submit multiple data files to ClickHouse database. The main implementation includes:

[0083] (1) Constructor: Accepts a FileReaderOption parameter, which contains configuration information, such as how to read files, database connection information, etc. In the constructor, a ClickhouseProxy object is initialized to interact with the ClickHouse database and obtain the metadata of the target database table.

[0084] (2) Submit operation: The commit method receives a List <ckfileaggcommitinfo>A list of (ClickHouse aggregate commit information collection), where each CKFileAggCommitInfo object represents a set of aggregated file information ready to be submitted to the database. The commit method iterates over this list internally, and for each CKFileAggCommitInfo, tries to append the file to the corresponding table in ClickHouse.

[0085] (3) Merge operation: The combine method is used to merge multiple CKFileCommitInfo (Clickhouse aggregate commit information) objects into one CKFileAggCommitInfo object. This process involves merging file information, that is, classifying all files according to the Shard to which they belong, which ensures that the data is more evenly distributed in ClickHouse and improves query efficiency.

[0086] (4) Rollback operation: In the current implementation, the abort method does not do anything, which usually means that when data submission fails, no special cleanup work is required, because ClickHouse itself will handle the unsuccessfully submitted data.

[0087] (5) Get the proxy: The getProxy method is used to get the ClickhouseProxy object. This is a thread-safe method that ensures the uniqueness of the ClickhouseProxy object in a multi-threaded environment, avoids unnecessary database connection creation, and saves resources.

[0088] (6) Release resources: The close method is used to release resources. When the ClickhouseFileSinkAggCommitter object is no longer used, this method should be called to close ClickhouseProxy and disconnect from the database.

[0089] (7) Attach files to ClickHouse: The attachFileToClickhouse method is one of the core methods in the entire class. It sends SQL commands to the database through ClickHouseRequest (ClickHouse database request) and uses the ALTER TABLE...ATTACH PART statement to attach the partition file in the file system to the corresponding partition of the ClickHouse table. This process physically moves the file data to the storage area of ​​the database to make the data available.

[0090] 3. JDBC single table submitter

[0091] JdbcSinkAggregatedCommitter is designed for JDBC connectors to handle transactional data commits. It is mainly used in distributed data stream processing systems as part of the JDBC Sink Connector to ensure data consistency and transaction integrity. The main implementation includes:

[0092] (1) Initialize the XA protocol components: Initialize the XaFacade and XaGroupOps objects, which encapsulate the XA interface of the JDBC connection and a set of XA operations respectively. XaFacade provides low-level interaction with the database, while XaGroupOps handles group commit and rollback of transactions.

[0093] (2) Open the connection: The tryOpen() method ensures that the XaFacade is open. If it is not open, it tries to open the connection, otherwise an exception is thrown. This is to ensure that the connection to the database is available before performing a commit or rollback operation.

[0094] (3) Commit operation: The commit() method receives a list of JdbcAggregatedCommitInfo (JDBC single-table aggregate committer), each of which contains a set of XidInfo (transaction identifiers). This method calls the commit() method of XaGroupOps to try to commit these transactions. If the commit fails, the commit() method returns a list of XidInfo that needs to be retried. This information will be encapsulated in a new JdbcAggregatedCommitInfo object and returned to the caller for retry.

[0095] (4) Merge operation: The combine() method combines a list of XidInfo into a JdbcAggregatedCommitInfo object for subsequent commit operations.

[0096] (5) Rollback operation: The abort() method receives a list of JdbcAggregatedCommitInfo and calls the rollback() method of XaGroupOps for each item to roll back the transaction. This is critical when the commit fails to maintain data consistency.

[0097] (6) Close the connection: After completing all commit and rollback operations, close XaFacade to release database connection resources.

[0098] 4. JDBC multi-table submitter

[0099] MultiTableSinkAggregatedCommitter (JDBC multi-table aggregate committer) implements a multi-table data commit aggregator that can manage multiple independent SinkAggregatedCommitter (aggregate write committer) instances, which are responsible for committing different table data. It supports data submission, aggregation, rollback and resource management, and also handles possible exceptions to ensure the robustness of data processing. The main implementation includes:

[0100] (1) Initialize the resource manager: receive a Map <String,SinkAggregatedCommitter<?,?> >(key-value pair of different types of aggregate committers, a data structure) type parameter, where the key is the identifier of the table and the value is the committer instance of the specific table. This allows the commit logic of different tables to be encapsulated together for unified management. The init() method initializes the MultiTableResourceManager (mixed table resource manager), and if the committers of all tables support multi-table transactions, it creates and sets the resource manager. This step is to ensure that the committers of all tables can work in the same transaction context, thereby achieving transaction consistency across tables.

[0101] (2) Commit operation: The commit() method receives a series of MultiTableAggregatedCommitInfo objects, which contain the commit information of each table. This method will perform a commit operation on the committer of each table and collect all error information that may occur during the commit process. If any commit fails, the failure information will be encapsulated in the MultiTableAggregatedCommitInfo object and returned for the upper-level logic to handle errors or retry.

[0102] (3) Merge operation: The combine() method is responsible for aggregating multiple MultiTableCommitInfo (mixed data table commit information) objects into a MultiTableAggregatedCommitInfo object. This process involves merging the commit information of different tables so that subsequent commit operations can process the data of all tables at once.

[0103] (4) Rollback operation: The abort() method implements the transaction rollback logic. When a commit operation fails, this method can be called to roll back the commit of all related tables to maintain data consistency.

[0104] (5) Release resources: The close() method is responsible for releasing the resources of all table submitters and resource managers to ensure that there is no resource leakage.

[0105] Based on the same inventive concept, the present invention also proposes a device for ensuring the consistency of distributed data collected. The implementation of the device can refer to the implementation of the above method, and the repeated parts will not be repeated. The term "module" used below can be a combination of software and / or hardware designed for a predetermined function. Although the device described in the following embodiments is preferably designed with software, the design of hardware, or a combination of software and hardware is also possible and conceivable.

[0106] Figure 5 Schematic diagram of the structure of the device for ensuring the consistency of distributed data collection in the present invention. Figure 5 As shown, the device comprises:

[0107] SeaCrabSink interface definition module 101 defines the creation methods of the target data writer, target data submitter, and target data aggregation submitter, and is responsible for the full life cycle management of the Sink component, including initialization, component construction, and serialization processing;

[0108] The target data writer interface module 102 carries the core logic of data writing and has writing, pre-commitment and snapshot state management methods for writing a single data element, preparing submission information and periodically saving state snapshots;

[0109] The target data submitter interface module 103 is responsible for the submission process. Its data submission method receives the submission information and attempts to pass it to the external data receiving end. It is also equipped with a data submission cancellation method to cope with the rollback requirement when the submission fails.

[0110] The target data aggregation submitter interface module 104, as an advanced version of the target data submitter interface, introduces a data integration method to integrate multiple submission information into a single aggregation information and then submit it.

[0111] The pre-commit method of the target data writer interface constitutes the first phase of the two-phase commit, and the data submission method of the target data committer interface or the target data aggregation committer interface constitutes the second phase of the two-phase commit;

[0112] Phase 1: The coordinator node sends a commit preparation request to all participant nodes and waits for their replies. After receiving the commit preparation request, each participant node executes data updates related to the distributed transaction and saves the operation results in the local log. If the participant node executes successfully, it returns a "complete" message to the coordinator node. When the coordinator node receives the return message from all participant nodes, the entire distributed transaction enters the second phase. If one participant node returns a failure, the coordinator node sends a rollback transaction request to all participant nodes, indicating that the distributed transaction execution fails.

[0113] Phase 2: The coordinator node decides whether to commit the transaction based on the responses from the participant nodes. If the coordinator node receives all responses agreeing to commit, it sends a commit request to all participant nodes and waits for their confirmation. After receiving the commit request, the participant node updates the operation result to the database and sends a confirmation message to the coordinator node. After receiving the confirmation messages from all participant nodes, the coordinator node finally decides whether to commit or roll back the transaction and notifies all participant nodes of the decision.

[0114] The data submission methods of the target data submitter and target data aggregation submitter interfaces are both idempotent.

[0115] It should be noted that although several modules of the device for ensuring the consistency of distributed data acquisition are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be concretized in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for concretization.

[0116] Based on the above invention concept, Figure 6 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, the aforementioned method for ensuring the consistency of distributed data collected is designed.

[0117] Based on the aforementioned inventive concept, the present invention further proposes a computer-readable storage medium, which stores a computer program for executing the aforementioned method for ensuring the consistency of distributed data in data collection.

[0118] The method and device for ensuring the consistency of distributed data collection proposed in the present invention have the following highlights:

[0119] 1. Enhanced data consistency: By ensuring the exact semantics of data, these components help reduce the risk of data duplication or loss and ensure data consistency and accuracy. In the field of distributed systems and data processing, "Exactly-Once Semantics" means that the system ensures that each operation or transaction is executed exactly once, neither more nor less.

[0120] 2. Improve data processing efficiency: By optimizing the process of data writing and submission, the speed and efficiency of data processing are improved, especially for large-scale data stream processing tasks.

[0121] 3. Simplify data management: Through integrated management and submission strategies, the data management process is simplified and development and maintenance costs are reduced.

[0122] 4. Enhanced system robustness: By supporting transactional commit or rollback operations, these components enhance the robustness of the system and maintain data consistency even when failures occur.

[0123] 5. Support for advanced data processing needs: Optimization for specific databases (such as ClickHouse) meets the needs of high-performance data processing, especially when fast query response is required.

[0124] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements contained in the spirit and scope of the attached claims.

[0125] Regarding the limitation of the protection scope of the present invention, those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.< / ckfileaggcommitinfo>

Claims

1. A method for ensuring the consistency of distributed data collection, characterized in that: The method includes: SeaCrabSink interface, as the core entry of SeaCrabSink framework, defines the creation methods of target data writer, target data submitter and target data aggregation submitter, and is responsible for the full life cycle management of Sink component, including initialization, component construction and serialization processing; The target data writer interface carries the core logic of data writing and has methods for writing, pre-committing, and snapshot status management. It is used for writing a single data element, preparing submission information, and periodically saving status snapshots. The target data submitter interface is responsible for the submission process. Its data submission method receives the submission information and attempts to pass it to the external data receiving end. It is also equipped with a data submission cancellation method to cope with the rollback requirements when the submission fails. The target data aggregation submitter interface, as an advanced version of the target data submitter interface, introduces a data integration method to integrate multiple submission information into a single aggregation information and then submit it.

2. The method for ensuring the consistency of distributed data acquisition according to claim 1, characterized in that: The pre-submission method of the target data writer interface constitutes the first stage of the two-stage submission, and the data submission method of the target data submitter interface or the target data aggregation submitter interface constitutes the second stage of the two-stage submission; Phase 1: The coordinator node sends a submission preparation request to all participant nodes and waits for their replies; After receiving the submit preparation request, each participant node executes the data update related to the distributed transaction and saves the operation results in the local log; if the participant node executes successfully, it returns a "completion" message to the coordinator node; When the coordinator node receives the return message from all the participant nodes, the entire distributed transaction enters the second stage; if one participant node fails to return, the coordinator node sends a rollback transaction request to all the participant nodes, which means that the distributed transaction execution fails; Phase 2: The coordinator node decides whether to commit the transaction based on the responses from the participant nodes. If the coordinator node receives all responses agreeing to commit, it sends a commit request to all participant nodes and waits for their confirmation. After receiving the commit request, the participant node updates the operation result to the database and sends a confirmation message to the coordinator node. After receiving the confirmation messages from all participant nodes, the coordinator node finally decides whether to commit or roll back the transaction and notifies all participant nodes of the decision.

3. The method for ensuring the consistency of distributed data acquisition according to claim 1, characterized in that: The data submission methods of the target data submitter and the target data aggregation submitter interface are both idempotent.

4. A device for ensuring the consistency of distributed data collection, characterized in that: The device includes: SeaCrabSink interface definition module, which defines the creation methods of target data writer, target data submitter and target data aggregation submitter, and is responsible for the full life cycle management of Sink components, including initialization, component construction and serialization processing; The target data writer interface module carries the core logic of data writing and has writing, pre-commitment and snapshot status management methods, which are used for writing a single data element, preparing submission information and periodically saving status snapshots; The target data submitter interface module is responsible for the submission process. Its data submission method receives the submission information and attempts to pass it to the external data receiving end. It is also equipped with a data submission cancellation method to cope with the rollback requirements when the submission fails. The target data aggregation submitter interface module, as an advanced version of the target data submitter interface, introduces a data integration method to integrate multiple submission information into a single aggregation information and then submit it.

5. The device for ensuring consistency of distributed data collection according to claim 4, characterized in that: The pre-submission method of the target data writer interface constitutes the first stage of the two-stage submission, and the data submission method of the target data submitter interface or the target data aggregation submitter interface constitutes the second stage of the two-stage submission; Phase 1: The coordinator node sends a submission preparation request to all participant nodes and waits for their replies; After receiving the submit preparation request, each participant node executes the data update related to the distributed transaction and saves the operation results in the local log; if the participant node executes successfully, it returns a "completion" message to the coordinator node; When the coordinator node receives the return message from all the participant nodes, the entire distributed transaction enters the second stage; if one participant node fails to return, the coordinator node sends a rollback transaction request to all the participant nodes, which means that the distributed transaction execution fails; Phase 2: The coordinator node decides whether to commit the transaction based on the responses from the participant nodes. If the coordinator node receives all responses agreeing to commit, it sends a commit request to all participant nodes and waits for their confirmation. After receiving the commit request, the participant node updates the operation result to the database and sends a confirmation message to the coordinator node. After receiving the confirmation messages from all participant nodes, the coordinator node finally decides whether to commit or roll back the transaction and notifies all participant nodes of the decision.

6. The device for ensuring consistency of distributed data collection according to claim 4, characterized in that: The data submission methods of the target data submitter and the target data aggregation submitter interface are both idempotent.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 3.

Images