Consistency data writing method and system, electronic device and storage medium

Abstract

Embodiments of the present application provide a consistent data writing method and system, electronic equipment and storage medium, which are applied to a connector of a data source or a connector of a destination. The connector of the data source is used to write data from the data source to Kafka, and the connector of the destination is used to write data from Kafka to the destination. First, it is determined whether the connector has transaction capability. If the connector has transaction capability, the data writing operation and the offset writing operation are submitted as a same transaction through the connector, so that it is determined that the transaction is successfully executed after the data writing operation and the offset writing operation are both completed. If the connector does not have transaction capability, it is determined that the data writing is completed after the data writing operation and the offset writing operation are both executed through the connector by using a two-phase commit protocol. By determining whether the connector has transaction capability and taking different measures, the consistency of data before and after data writing is ensured.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and particularly relates to a consistent data writing method, system, electronic device, and storage medium. Background Technology

[0002] With the rapid development of information technology, data acquisition has been widely applied in the Internet and distributed fields. In actual enterprise data application scenarios, building a data acquisition system faces multiple challenges. Because the data source and destination types may differ, the standards for defining a field type vary, and the data source is constantly changing, and the persistence of data bias and data transmission are asynchronous, bias and data position misalignment can easily occur, affecting data consistency. Currently, data acquisition often uses the Message Queue (MQ) mode. In this mode, independent data source connectors and destination connectors are needed to complete the input and output of data to the MQ. Kafka Connect is based on the MQ mode, which only supports end-to-end at least once consistency guarantees. In this mode, if deduplication is not performed through primary keys or downstream programs, data inconsistency can occur before and after data is written. Summary of the Invention

[0003] In view of this, one of the technical problems solved by the embodiments of the present invention is to provide a consistent data writing method, system, electronic device and storage medium to overcome or avoid the above problems as much as possible.

[0004] To achieve the above objectives, this invention provides a consistent data writing method applied to a connector for a data source or a connector for a destination. The connector for the data source is used to write data from the data source to Kafka, and the connector for the destination is used to write data from Kafka to the destination. First, it is determined whether the connector has transaction capabilities. If it does, the data writing operation and the bias writing operation are committed as a single transaction through the connector, so that the transaction is considered successful after both the data writing operation and the bias writing operation are completed. Alternatively, if it does not have transaction capabilities, the data writing is considered complete after the data writing operation and the bias writing operation are completed using a two-phase commit protocol through the connector.

[0005] Optionally, after the data is written from the data source to Kafka, based on the data processing flow built through Kafka Streams, the data and the corresponding bias are read from Kafka, the data is transformed according to the logic corresponding to the data processing flow, and the transformed data and the bias are rewritten to Kafka.

[0006] Optionally, the data write operation and the bias write operation can be committed as a single transaction. After both operations are completed, the transaction is considered successful. This includes: committing the data write operation and the bias write operation as a single transaction; and after confirming the bias write operation's success, sending a message indicating successful execution to Kafka or the destination to confirm transaction success. Alternatively, after confirming successful transaction execution, or after confirming data writing is complete, storing the bias corresponding to the successfully written data in a pre-defined bias table.

[0007] Optionally, before determining whether the connector has transaction capabilities, it is determined whether the data has a primary key for unique identification. If so, the step of determining data write completion after both the data write operation and the bias write operation have been executed using the two-phase commit protocol is performed. Determining data write completion after both the data write operation and the bias write operation have been executed using the two-phase commit protocol includes: in the preparation phase, submitting preparation indication information for both the data write operation and the bias write operation, and receiving preparation success feedback data corresponding to the data write operation or the bias write operation; after confirming that both the preparation success feedback data corresponding to the data write operation and the bias write operation have been successfully received, submitting execution indication information for the data write operation and the bias write operation, and receiving execution success feedback data for the data write operation and the bias write operation, and then determining data write completion. If receiving preparation success feedback data or execution success feedback data corresponding to the data write operation or the bias write operation fails, a rollback operation is performed in the preparation phase.

[0008] This invention also provides a consistent data writing system, including: a data source, a connector applied to the data source, a connector applied to the destination, a Kafka, and a destination. The connector for the data source is used to write data from the data source to the Kafka, and the connector for the destination is used to write data from the Kafka to the destination. The connector is used to determine whether it has transaction capabilities. If it does, the connector commits the data write operation and the bias write operation as a single transaction, and the transaction is considered successful after both the data write operation and the bias write operation are completed. Alternatively, if it does not have transaction capabilities, the connector uses a two-phase commit protocol to determine that the data write is complete after both the data write operation and the bias write operation are completed.

[0009] This invention also provides an electronic device, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the method described above.

[0010] This invention also provides a computer storage medium storing a computer program that, when executed by a processor, implements the methods described in the above embodiments.

[0011] As can be seen from the above technical solutions, the embodiments of the present invention determine whether the connectors for writing data from the data source to Kafka and the connectors for writing data from Kafka to the destination have transaction capabilities. If they do, the data writing operation and the bias writing operation are committed as a single transaction through the connector to ensure the correspondence between the data and the bias, prevent data omissions or duplicate writes, and ensure consistent data writing to Kafka or the destination. The transaction is considered successful after both the data writing operation and the bias writing operation are completed. Alternatively, if the connectors do not have transaction capabilities, the two-phase commit protocol is used to determine that the data writing is complete after both the data writing operation and the bias writing operation are completed. The two-phase commit protocol prevents the data source or Kafka from resending a batch of data due to bias persistence failure, which could lead to data duplication during the writing process. By determining whether the connectors have transaction capabilities and taking different countermeasures, the problem of data inconsistency before and after data writing is solved. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0013] Figure 1 This is a schematic diagram of the consistent data writing system according to Embodiment 1 of the present invention;

[0014] Figure 2 The application of Embodiment 2 of the present invention Figure 1 Flowchart of the connector method;

[0015] Figure 3 This is a flowchart of data processing in Kafka according to Embodiment 3 of the present invention;

[0016] Figure 4This is a flowchart of the consistent data writing method according to Embodiment 4 of the present invention;

[0017] Figure 5 This is a flowchart of the consistent data writing method according to Embodiment 5 of the present invention;

[0018] Figure 6 This is a schematic diagram of the structure of an electronic device according to Embodiment Six of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0020] It should be noted that implementing any technical solution of the embodiments of the present invention does not necessarily require achieving all of the above advantages simultaneously. The specific implementation of the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0021] Example 1

[0022] Figure 1 This is a schematic diagram of a consistent data writing system according to an embodiment of the present invention. The consistent data writing system provided in this embodiment includes: a data source, a connector applied to the data source, a connector applied to the destination, a Kafka, and a destination. The connector of the data source is used to write data from the data source to Kafka, and the connector of the destination is used to write data from Kafka to the destination. The connector is used to determine whether it has transaction capabilities. If it has transaction capabilities, the connector commits the data writing operation and the bias writing operation as a single transaction, and the transaction is considered successful after both the data writing operation and the bias writing operation are completed. Alternatively, if it does not have transaction capabilities, the connector uses a two-phase commit protocol to determine that the data writing is complete after both the data writing operation and the bias writing operation are completed.

[0023] Specifically, Kafka is a high-throughput distributed publish-subscribe messaging system with features such as system decoupling, redundant storage, traffic shaping, buffering, asynchronous communication, and scalability. As a data processing platform between the data source and the destination, after the data source connector writes data from the data source to Kafka, Kafka persists the data until it has been fully processed, thereby avoiding the risk of data loss.

[0024] Specifically, Kafka connectors are tools used for scalable and reliable streaming of data between Kafka and other systems. They enable the rapid writing and moving of large amounts of data out of Kafka. A Kafka connector instance is a logical job responsible for managing data replication between Kafka and another system. All classes implemented or used by a connector are defined in connector plugins; both connector instances and connector plugins can be referred to as "connectors." Kafka connectors include connectors applied to data sources and connectors applied to destinations. Data source connectors are used to write data from the data source to Kafka, and destination connectors are used to write data from Kafka to the destination.

[0025] Specifically, transaction capability means that a transaction is an indivisible unit of work. All operations within a transaction either occur or none occur, and data integrity and business logic consistency are not compromised before or after the transaction begins. A single transaction does not affect the operation of other transactions. For most relational databases, such as Oracle, MySQL, and Microsoft SQL Server, the connectors used to transmit their data possess transaction capability.

[0026] Specifically, bias refers to the position information of a data element in the data. It can be used as a positioning mark of the data element. During the data writing process, it can record the last position of the current data being written. This can prevent new data from being added to a certain set of data during the writing process, which would cause a mismatch between the data and the corresponding bias. Committing the data and bias writing operations as the same transaction can prevent data from being missed or duplicated.

[0027] For example, data writing and bias writing rely on different transmission media. If data is written first and then the bias is committed, and if the data writing is successful but the bias commit fails, the bias obtained by Kafka or the destination will still be the previous bias, resulting in duplicate data writing. If the bias is committed first and then the data is written, and if the bias commit is successful but the data writing fails, the bias obtained by Kafka or the destination will already be the new bias, resulting in missing data writing.

[0028] Specifically, the two-phase commit protocol is an algorithm designed to maintain consistency during transaction commit. When committing a transaction, either all participating processes commit the transaction or all processes cancel the transaction, thereby ensuring that both data write operations and bias write operations are completed.

[0029] Preferably, the process of writing data from the data source to Kafka and from Kafka to the destination should introduce some boundary conditions to mark the end of the writing of the same batch of data. For example, at the beginning and end of each batch of sending, control signals are introduced at the beginning and end of the same batch of data to indicate whether the data is from the same batch. Data and bias are written by batch to ensure data continuity.

[0030] As can be seen from the above technical solutions, the embodiments of the present invention determine whether the connectors for writing data from the data source to Kafka and the connectors for writing data from Kafka to the destination have transaction capabilities. If they do, the data writing operation and the bias writing operation are committed as a single transaction through the connector to ensure the correspondence between the data and the bias, prevent data omissions or duplicate writes, and ensure consistent data writing to Kafka or the destination. The transaction is considered successful after both the data writing operation and the bias writing operation are completed. Alternatively, if the connectors do not have transaction capabilities, the two-phase commit protocol is used to determine that the data writing is complete after both the data writing operation and the bias writing operation are completed. The two-phase commit protocol prevents the data source or Kafka from resending a batch of data due to bias persistence failure, which could lead to data duplication during the writing process. By determining whether the connectors have transaction capabilities and taking different countermeasures, the problem of data inconsistency before and after data writing is solved.

[0031] Example 2

[0032] Figure 2 For application Figure 1 The flowchart of the connector method in this embodiment provides a consistent data writing method, applied to a data source connector or a destination connector. The data source connector is used to write data from the data source to Kafka, and the destination connector is used to write data from Kafka to the destination. First, it is determined whether the connector has transaction capabilities. If it does, the data write operation and the bias write operation are committed as a single transaction through the connector, so that the transaction is considered successful after both the data write operation and the bias write operation are completed. Alternatively, if it does not have transaction capabilities, the data write is considered complete after the data write operation and the bias write operation are completed using a two-phase commit protocol through the connector.

[0033] It should be noted that the specific working process of the method described above can be referred to the corresponding process description in the foregoing system embodiments, and will not be repeated here.

[0034] Example 3

[0035] like Figure 3This is a data processing flowchart in Kafka according to Embodiment 3 of the present invention. In this embodiment, after the data is written from the data source to Kafka, the method further includes: reading data and the corresponding bias from Kafka based on the data processing flow constructed by Kafka Streams, performing logical transformation on the data corresponding to the data processing flow, and rewriting the transformed data and the bias into Kafka.

[0036] Specifically, Kafka Streams is a real-time stream computing library. Stream computing is the process of continuously merging new data to compute results. Kafka Streams can perform logical transformations such as data cleaning and transformation on out-of-order and late-arriving data. Typically, historical and real-time data are written to Kafka simultaneously. Kafka Streams can effectively analyze large-scale, constantly changing data in real time, capturing potentially useful information and processing it accordingly. Furthermore, the process of Kafka Streams reading data and its corresponding biases from Kafka is unrestricted and has no predetermined start or end. The transformed results can be streamed back to Kafka, ensuring data consistency before and after processing.

[0037] In this embodiment, based on the data processing flow built with Kafka Streams, data and its corresponding bias are read from Kafka, and the data is transformed according to the logic of the data processing flow. Then, the transformed data and the bias are rewritten into Kafka. Kafka Streams is used to complete the data fusion task of batch data in Kafka, ensuring the consistency of data before and after data processing.

[0038] Example 4

[0039] like Figure 4 This is a flowchart of the consistent data writing method according to Embodiment 4 of the present invention. In this embodiment, the data writing operation and the bias writing operation are committed as a single transaction. After both the data writing operation and the bias writing operation are completed, the transaction is determined to have been successfully executed. This includes: committing the data writing operation and the bias writing operation as a single transaction; after determining that the bias writing operation has been successfully executed, sending a message indicating successful execution of the data writing operation to Kafka or the destination, thus confirming successful transaction execution. Specifically, after confirming successful transaction execution, or after confirming that the data writing is complete, the bias corresponding to the successfully written data is stored in a pre-set bias table.

[0040] Specifically, by committing the data write operation and the bias write operation as a single transaction, the downstream will not see the sent data until the bias is successfully updated. The bias, as a positioning marker for data elements, records the last position of the data being written during the data writing process. This prevents new data from being added to a certain set of data during the writing process, which could cause a mismatch between the data and the corresponding bias. Committing the data write operation and the bias write operation as a single transaction can prevent data omissions or duplicate data writes. After confirming that the bias write operation has been executed successfully, a message indicating that the data write operation has been successfully executed is sent to Kafka or the destination, confirming that the transaction has been successfully executed.

[0041] Preferably, after determining that the transaction has been successfully executed or that the data writing has been completed, the bias corresponding to the successfully written data is stored in a pre-set bias table to record the last position of the current data writing, thus ensuring the correspondence between the data and the bias.

[0042] In this embodiment, by committing the data write operation and the bias write operation as the same transaction, the transaction is confirmed to be successful after the bias write operation is confirmed to be successful, thus preventing data omissions or duplicate data writes. At the same time, after confirming the successful execution of the transaction or the completion of data writing, the bias corresponding to the written data is stored in a pre-set bias table, ensuring the matching relationship between the written data and the corresponding bias, and ensuring the consistency of data before and after data processing.

[0043] Example 5

[0044] like Figure 5 The flowchart below shows the consistent data writing method according to Embodiment 5 of the present invention. In this embodiment, before determining whether the connector has transaction capability, the method further includes: determining whether the data has a primary key used as a unique identifier; if so, executing a step of determining that the data writing is complete after both the data writing operation and the bias writing operation are completed using the two-phase commit protocol.

[0045] During the preparation phase, preparation indication information for data write operation and bias write operation is submitted respectively, and preparation success feedback data corresponding to data write operation or bias write operation is received.

[0046] After confirming that the preparation success feedback data corresponding to both the data write operation and the bias write operation have been successfully received, the execution instruction information for the data write operation and the bias write operation is submitted. After receiving the execution success feedback data for both the data write operation and the bias write operation, the data write is considered complete. If receiving the preparation success feedback data or the execution success feedback data corresponding to the data write operation or the bias write operation fails, the preparation phase is entered to perform a rollback operation.

[0047] Specifically, a primary key is used to uniquely identify a row of data. Its characteristics include being non-repeating and non-nullable. Therefore, primary keys can be used to delete duplicate data, ensuring data uniqueness during data transmission. Since a primary key uniquely identifies a record, it ensures that data updates and deletions will not result in misattribution, guaranteeing a match between the data and its corresponding bias. If the data has a primary key that serves as a unique identifier, a two-phase commit protocol is used to ensure that the data and its bias are written to Kafka or the destination.

[0048] Specifically, the two-phase commit protocol is an algorithm designed to maintain consistency when committing transactions. It can be introduced into the Kafka end of the data source connector or the destination end of the destination connector. The two-phase commit protocol includes a preparation phase and a commit phase.

[0049] During the preparation phase, the coordinator sends data and biases to two different participants. The coordinator acts as the transaction coordinator and manager when using the two-phase commit protocol. It communicates with each participant to manage the transaction within the two-phase commit protocol. The participants manage all transaction resources; for example, in this application, the participants manage the data and biases and have the ability to commit and rollback. After sending the data and biases to the two different participants, the coordinator inquires whether the transaction commit operation can be performed and then waits for the participants' responses. The participants prepare for the transaction commit and send a response to the coordinator's transaction inquiry. During the commit phase, the coordinator sends a commit request for the data and biases to all participants. After receiving the commit request from the coordinator, the participants formally execute the transaction commit operation, release the transaction resources occupied during the entire transaction execution, and send a commit confirmation message to the coordinator. After receiving confirmation messages from all participants that the data write operation and bias write operation were successfully executed, the coordinator confirms that the data write is complete.

[0050] If any participant reports a failure to the coordinator, or if the coordinator does not receive feedback from all participants after a timeout (i.e., failure to receive preparation success feedback or execution success feedback data corresponding to the data write operation or bias write operation), the transaction will be interrupted and a rollback operation will be performed. The coordinator will send a rollback request to all participants. After all participants send confirmation of the rollback request, the coordinator will resend the data and the bias to two different participants to continue the preparation phase tasks until both the data write operation and the bias write operation are completed. After the rollback is completed, the resources occupied during the entire transaction execution will be released.

[0051] In this embodiment, primary key deduplication can ensure the uniqueness of data during data transmission; the two-phase commit protocol ensures that data writing and bias writing either commit the transaction or cancel the transaction, thereby ensuring that data writing and bias writing operations can be completed synchronously, thus guaranteeing the consistency of data before and after data writing.

[0052] Example 6

[0053] Figure 6 This is a schematic diagram of the structure of an electronic device according to Embodiment 5 of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0054] The electronic device may include: a processor 602, a communications interface 604, a memory 606, and a communications bus 608.

[0055] in:

[0056] The processor 602, communication interface 604, and memory 606 communicate with each other via communication bus 608.

[0057] Communication interface 604 is used for communication with other electronic devices or servers.

[0058] The processor 602 is used to execute program 610, specifically to perform the relevant steps in the above method embodiments.

[0059] Specifically, program 610 may include program code that includes computer operation instructions.

[0060] The processor 602 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0061] Memory 606 is used to store program 610. Memory 606 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0062] Specifically, program 610 can be used to cause processor 602 to perform the operation corresponding to the method described in any of the foregoing multiple method embodiments.

[0063] The specific implementation of each step in program 610 can be found in the corresponding steps and units described in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0064] Finally, this application embodiment also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the operation corresponding to any of the methods in the above-described multiple method embodiments.

[0065] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0066] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0067] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0068] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A method of writing data coherently, characterized by, A connector applied to a data source or a destination, wherein the data source connector is used to write data from the data source to Kafka, and the destination connector is used to write data from Kafka to the destination, the method comprising: Determine whether the data has a primary key for unique identification. If it does, execute the step of determining that the data writing is complete after both the data write operation and the bias write operation have been completed using the two-phase commit protocol. Determine whether the connector has transaction capabilities; If transaction capability is available, the data write operation and the bias write operation are committed as a single transaction through the connector, so that the transaction is considered to have been successfully executed after both the data write operation and the bias write operation are completed. If transaction capability is not available, the connector is used to execute the following steps: after both the data write operation and the bias write operation have been completed, the data write is confirmed to be complete. The execution of the two-phase commit protocol, specifically the step of determining data writing completion after both the data write operation and the bias write operation have been executed, includes: in the preparation phase, submitting preparation indication information for the data write operation and the bias write operation respectively, and receiving preparation success feedback data corresponding to the data write operation or the bias write operation; after confirming that both the data write operation and the preparation success feedback data corresponding to the bias write operation have been successfully received, submitting execution indication information for the data write operation and the bias write operation, and receiving execution success feedback data for the data write operation and the bias write operation, and then determining data writing completion; if receiving preparation success feedback data or execution success feedback data corresponding to the data write operation or the bias write operation fails, then entering the preparation phase for a rollback operation; After the data is written from the data source to the Kafka, the method further includes: based on the data processing flow built through Kafka Streams, reading the data and the bias corresponding to the data from the Kafka, performing logical transformation on the data corresponding to the data processing flow, and then rewriting the transformed data and the bias back into the Kafka; After determining that the transaction was successfully executed, or that the data writing was completed, the method further includes: storing the bias corresponding to the successfully written data into a pre-set bias table.

2. The method of claim 1, wherein, The step of committing the data write operation and the bias write operation as a single transaction, and determining the successful execution of the transaction after both the data write operation and the bias write operation are completed, includes: The data write operation and the bias write operation are committed as the same transaction; After confirming that the bias write operation was executed successfully, a message indicating that the data write operation was successfully executed is sent to the Kafka or the destination to confirm that the transaction was executed successfully.

3. A consistent data write system, comprising: Data source, connector applied to the data source, connector applied to the destination, Kafka, destination, wherein the connector for the data source is used to write data from the data source to the Kafka, and the connector for the destination is used to write data from the Kafka to the destination; The connector is used to determine whether it has transaction capabilities. If it does, the connector is used to commit the data write operation and the bias write operation as a single transaction, so that the transaction is considered successful after both the data write operation and the bias write operation are completed. If it does not have transaction capabilities, the connector is used to determine that the data write is complete after both the data write operation and the bias write operation are completed using a two-phase commit protocol. The step of determining data writing completion by utilizing a two-phase commit protocol to confirm completion after both the data write operation and the bias write operation have been executed includes: in the preparation phase, submitting preparation indication information for both the data write operation and the bias write operation, and receiving preparation success feedback data corresponding to the data write operation or the bias write operation; after confirming that both the preparation success feedback data for the data write operation and the bias write operation have been successfully received, submitting execution indication information for both the data write operation and the bias write operation, and receiving execution success feedback data for both the data write operation and the bias write operation, then determining data writing completion; if receiving preparation success feedback data or execution success feedback data for the data write operation or the bias write operation fails, then entering the preparation phase for a rollback operation; Wherein, after the data is written from the data source to the Kafka, the connector is further configured to: read the data and the corresponding bias from the Kafka based on the data processing flow constructed through KafkaStreams; perform logical transformation on the data corresponding to the data processing flow; and rewrite the transformed data and the bias into the Kafka; after determining that the transaction has been successfully executed, or after determining that the data writing has been completed, store the bias corresponding to the successfully written data into a pre-set bias table; Before the connector is used to determine whether it has transaction capabilities, it is determined whether the data has a primary key for unique identification. If it does, the step of using the two-phase commit protocol to determine that the data writing operation and the bias write operation have been completed is executed, and the data writing is completed.

4. An electronic device, comprising: include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in any one of claims 1-2.

5. A computer storage medium, characterized in that The computer storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-2.

Images