Data synchronization method and apparatus, electronic device, and storage medium

Abstract

The application provides a data synchronization method and device, electronic equipment and a storage medium, the method comprising: obtaining a data synchronization job to be executed, the data synchronization job being configured with a source database and an object to be synchronized; in the case that there is an open logical replication slot in the source database, merging the data synchronization job with a running job associated with the logical replication slot to generate a merged synchronization job; executing the merged synchronization job to obtain a logical log generated in the source database for the object to be synchronized through the logical replication slot; and synchronizing data obtained based on the logical log to a target database, thereby solving the problems of high load caused by frequent table scanning and high management overhead caused by triggers in the traditional scheme, while performing change data capture with high real-time performance and high accuracy, automatically optimizing the merged job in the background to avoid repeated extraction, effectively reducing the running load and resource consumption of the source database, and improving the flexibility and overall efficiency of data synchronization.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a data synchronization method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of the Internet and the Internet of Things (IoT) industries, databases such as GBase 8c are widely used in industries with stringent security and stability requirements, such as finance and telecommunications. Change data capture technology, as a core means of real-time data acquisition, is an important foundation for data synchronization, real-time analysis, and the construction of real-time data warehouses.

[0003] Existing real-time data acquisition solutions for GBase8c have significant drawbacks. Timestamp comparison solutions require polling and scanning the entire table, compromising real-time performance and significantly increasing database load in large-scale data scenarios. While trigger-based acquisition solutions can guarantee real-time performance, a large number of triggers consume additional database resources, interfering with normal business efficiency. Furthermore, in multi-table acquisition and high-concurrency write scenarios, existing solutions lead to severe resource waste and high synchronization latency, failing to meet the high-efficiency data synchronization requirements of demanding scenarios. Summary of the Invention

[0004] This invention provides a data synchronization method, apparatus, electronic device, and storage medium, which are particularly suitable for scenarios such as change data capture and real-time data warehouse construction in Gbase8c databases. By reusing logical replication slots and logical log parsing, it significantly reduces the performance consumption of the source database while ensuring low-latency and high-reliability real-time data acquisition.

[0005] This invention provides a data synchronization method, applied at a synchronization end, comprising: Obtain a data synchronization job to be executed, wherein the data synchronization job is configured with a source database and objects to be synchronized; If an open logical replication slot exists in the source database, the data synchronization job is merged with the running job associated with the logical replication slot to generate a merge synchronization job. Execute the merge and synchronization job to obtain the logical logs generated in the source database for the object to be synchronized through the logical replication slot; The data obtained from parsing the logical logs will be synchronized to the target database.

[0006] According to a data synchronization method provided by the present invention, merging the data synchronization job with the running job associated with the logical replication slot to generate a merged synchronization job includes: Get the set of subscribed objects corresponding to the running job; By comparing the object to be synchronized with the set of subscribed objects, an incremental object to be synchronized is obtained; While the logical replication slot is running continuously, the incremental objects to be synchronized are appended to the set of subscribed objects to obtain the merge synchronization job.

[0007] According to a data synchronization method provided by the present invention, the step of obtaining the data synchronization job to be executed includes: It displays a visual configuration interface that includes source configuration controls, destination configuration controls, and rule configuration controls; The system receives an assembly command triggered by operating the source configuration control, the destination configuration control, and the rule configuration control in the visual configuration interface. The assembly command carries a connection attribute associated with the source database, an object identifier associated with the object to be synchronized, and data processing rules. The assembly instructions are parsed and converted into code to obtain the data synchronization operation.

[0008] According to a data synchronization method provided by the present invention, the step of executing the merge synchronization job to obtain the logical log generated in the source database for the object to be synchronized through the logical replication slot further includes: Send a release instruction to the source database for the object to be synchronized, so as to generate a data change release set associated with the object to be synchronized in the source database; Through the merge synchronization operation, a connection based on the logical replication slot is established between the synchronization end and the data change release set, and a full data snapshot of the object to be synchronized is obtained through the connection.

[0009] According to a data synchronization method provided by the present invention, the step of sending a release instruction for the object to be synchronized to the source database to generate a data change release set associated with the object to be synchronized within the source database includes: Obtain the target data operation type that matches the object to be synchronized; Generate an object identifier that includes the object to be synchronized and a constraint configuration instruction for the target data operation type, as the release instruction; The release instruction is sent to the source database, so that the source database constructs a data change release set based on change records that conform to the target data operation type.

[0010] According to a data synchronization method provided by the present invention, the step of executing the merge synchronization job to obtain the logical log generated in the source database for the object to be synchronized through the logical replication slot includes: Determine the initial log position corresponding to the full data snapshot in the source database; The logical log is obtained through the logical replication slot; the logical log is obtained by reverse engineering the write-ahead log generated in the source database after the initial log position.

[0011] According to a data synchronization method provided by the present invention, the logical log is obtained by the source database by reversing the write-ahead log when it detects a data change event for the object to be synchronized.

[0012] According to a data synchronization method provided by the present invention, the step of synchronizing data obtained based on the parsed logical log to a target database includes: Based on the data processing rules, the logical log is processed to obtain a data stream in the target format; The target format data stream is routed to a destination write plugin that matches the type of the target database, and the target format data stream is written to the target database through the destination write plugin.

[0013] The present invention also provides a data synchronization device, applied at a synchronization end, comprising: The acquisition unit is used to acquire a data synchronization job to be executed, wherein the data synchronization job is configured with a source database and an object to be synchronized; The merging unit is used to merge the data synchronization job with the running job associated with the logical replication slot when there is an open logical replication slot in the source database, to generate a merged synchronization job. An execution unit is used to execute the merge synchronization job to obtain the logical logs generated in the source database for the object to be synchronized through the logical replication slot; The synchronization unit is used to synchronize the data obtained from parsing the logical log to the target database.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the data synchronization method as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data synchronization method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the data synchronization method as described above.

[0017] The data synchronization method, apparatus, electronic device, and storage medium provided by this invention acquire a data synchronization job configured with a source database and an object to be synchronized. When an open logical replication slot exists in the source database, the job is automatically merged with the running job to generate a merge synchronization job. Logical logs are then obtained through reused logical replication slots, and the parsed data is synchronized to the target database. This solves the problems of high load caused by frequent table scans and high management overhead caused by triggers in traditional solutions. By utilizing the logical replication characteristics and replication slot reuse mechanism of the database itself, change data is captured with high real-time performance and high accuracy. The merge job is automatically optimized in the background to avoid duplicate extraction, effectively reducing the operating load and resource consumption of the source database, improving the flexibility and overall efficiency of data synchronization, and meeting the stringent requirements of complex scenarios such as real-time data warehouses. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the data synchronization method provided by the present invention; Figure 2 This is a flowchart of the overall data synchronization method provided by the present invention; Figure 3 This is an example diagram of the log reverse decoding process provided by the present invention; Figure 4 This is an example diagram of the logical log provided by the present invention; Figure 5 This is a schematic diagram of the data synchronization device provided by the present invention; Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] With the booming development of emerging industries such as the Internet and the Internet of Things, real-time data has become a key factor determining enterprise competitiveness, and building a real-time data warehouse is an essential step towards data-driven decision-making. In the fields of database technology, real-time data acquisition, and real-time data warehouse construction, domestic databases such as Gbase8c have broad application prospects in industries with stringent requirements for data security, stability, and processing performance, such as finance and telecommunications. CDC (Change Data Capture) technology, as a key technology in the database field, focuses on accurately and efficiently capturing data change information in databases, covering changes caused by various operations from data insertion and updating to deletion, providing a data foundation for subsequent complex tasks such as data synchronization, real-time analysis, and the construction of real-time data warehouses.

[0022] However, current real-time data acquisition solutions for databases such as Gbase8c have the following main drawbacks: Firstly, there are timestamp comparison acquisition solutions. This approach introduces timestamp fields into the database table design, and the data acquisition program periodically polls each table to be monitored to filter changed data based on the timestamp field. This method relies solely on periodic polling and cannot guarantee real-time data performance. Furthermore, when dealing with large datasets, frequent table scans can place a heavy load on the database server, leading to overall performance degradation. Inappropriate timestamp precision settings can also cause data acquisition delays or duplicate collections. Secondly, there are trigger-assisted acquisition solutions. This approach creates corresponding triggers on key tables to push changed data. While this solution can guarantee real-time data performance, excessive trigger settings increase database overhead and affect the efficiency of normal business processing. In addition, when dealing with acquisition tasks across multiple tables in the same database or high-concurrency write scenarios, existing solutions often execute data acquisition operations independently, failing to fully utilize the computing power of the hardware platform and easily causing serious resource waste and extremely high latency.

[0023] To address this, the present invention provides a data synchronization method that leverages the logical replication characteristics of databases. Upon detecting an open logical replication slot in the source database, it automatically merges the data synchronization job to be executed with the running job. By reusing the logical replication slot to obtain the logical logs generated for the object to be synchronized, it performs data parsing and synchronization. This achieves zero-operation on database storage, avoiding the heavy load and operational overhead caused by frequent table scans and excessive triggers. It effectively eliminates the redundant extraction and resource waste caused by multiple jobs running independently in the same database. While reducing resource consumption and maintenance costs, it achieves low-latency, highly reliable real-time data acquisition and consistent synchronization.

[0024] Figure 1This is a flowchart illustrating the data synchronization method provided by the present invention. This method is applied to the synchronization end, such as a server with a data acquisition program, a real-time data processing channel, or a scheduling system deployed thereon. Figure 1 As shown, the method includes: Step 110: Obtain the data synchronization job to be executed. The data synchronization job is configured with the source database and the objects to be synchronized. Step 120: If there are enabled logical replication slots in the source database, merge the data synchronization job with the running job associated with the logical replication slot to generate a merge synchronization job. Step 130: Execute the merge synchronization job to obtain the logical logs generated in the source database for the objects to be synchronized through the logical replication slot; Step 140: Synchronize the data obtained from parsing the logical logs to the target database.

[0025] Specifically, considering that current data synchronization suffers from high resource consumption and high latency when facing large-scale data volumes and stringent real-time requirements, in order to effectively solve this problem and adapt to domestic databases, such as the Gbase8c architecture, the primary consideration is to utilize the underlying logical replication features of the database for real-time data synchronization.

[0026] Understandably, before performing data synchronization, it is necessary to first determine the data synchronization job to be executed. This data synchronization job is a task created to migrate or synchronize data from the source to the destination in real time. This data synchronization job must be configured with at least a source database and objects to be synchronized. The source database is the library from which the data originates, such as a Gbase8c database that requires real-time synchronization. The objects to be synchronized are the objects in the source database whose changes need to be monitored and captured, such as specific database tables, specific columns within tables, or a range of data. This embodiment of the invention does not specifically limit these.

[0027] Furthermore, after determining the data synchronization job, to avoid resource waste, the synchronization end will perform an optimization judgment on logical replication before the job starts, that is, to determine whether there are open logical replication slots in the source database. Logical replication slots are a mechanism in the database used to ensure that the master database will not delete the corresponding logs, such as WAL (Write-Ahead Logging) logs, before the subscriber confirms receipt, thereby ensuring data stream replication.

[0028] In practice, job configuration is typically done at the table level. This can lead to the same source database, such as different tables within a single Gbase8c database instance, being configured as multiple independent data synchronization jobs. If different logical replication slots are used during job execution, the limited number of logical replication slots supported by each database instance and the increased resource overhead and load on the database server due to the large number of slots enabled can exacerbate the problem. Therefore, in this embodiment of the invention, once an already enabled logical replication slot is identified in the source database, the data synchronization job is merged and optimized with the running job associated with that logical replication slot to generate a merged synchronization job. This allows multiple jobs targeting different tables or different objects to be synchronized in the same source database to reuse the same existing logical replication slot, significantly reducing the consumption of source database resources.

[0029] Subsequently, the synchronization end will execute the merge synchronization job. During this process, the synchronization end establishes a data flow connection with the source database through a reused logical replication slot and obtains the logical logs generated in the source database for the objects to be synchronized. Unlike traditional scheduled table scans or full extraction, logical replication has minimal impact on the performance of the source database during the replication process. It does not require direct data table scans of the database storage. Instead, the source database generates logs related to changes in the objects to be synchronized in the background, such as logs recording data changes caused by DML (Data Manipulation Language) operations. The synchronization end only needs to subscribe to and obtain these incremental logical logs through the logical replication slot. This greatly reduces the I / O and CPU (Central Processing Unit) resource consumption of the source database, making it very suitable for business scenarios such as finance and telecommunications where the performance and stability of the source database are critical.

[0030] Finally, the synchronization endpoint synchronizes the data obtained from the logical log parsing to the target database. That is, the acquired logical logs are parsed and transformed to restore them to a human-readable or destination-recognizable line-by-line data stream or structured data, and this data is written to a pre-specified destination, such as a real-time data warehouse, cache, message queue, or other type of target database. Here, through this read-write separation mechanism based on logical replication, data consistency in the source database is maintained while a low-latency real-time data channel is built, significantly improving data processing performance.

[0031] The data synchronization method provided by this invention obtains a data synchronization job configured with a source database and an object to be synchronized. When there is an enabled logical replication slot in the source database, the method automatically merges the job with the running job to generate a merge synchronization job. Then, it obtains logical logs through the reused logical replication slots and synchronizes the parsed data to the target database. This solves the problems of high load caused by frequent table scans and high management overhead caused by triggers in traditional solutions. By utilizing the logical replication characteristics and replication slot reuse mechanism of the database itself, the method captures change data with high real-time performance and high accuracy, while automatically optimizing the merge job in the background to avoid duplicate extraction. This effectively reduces the operating load and resource consumption of the source database, improves the flexibility and overall efficiency of data synchronization, and meets the stringent requirements of complex scenarios such as real-time data warehouses.

[0032] Based on the above embodiments, in step 120, the data synchronization job is merged with the running job associated with the logical replication slot to generate a merged synchronization job, including: Get the collection of subscribed objects corresponding to the running job; Compare the objects to be synchronized with the set of subscribed objects to obtain the incremental objects to be synchronized; While the logical replication slot is running continuously, the incremental objects to be synchronized are appended to the set of subscribed objects to obtain the merge synchronization job.

[0033] Specifically, the process of merging the data synchronization job with the running job associated with the logical replication slot can include: First, it's necessary to obtain the set of subscribed objects corresponding to the running job. In real-world scenarios, for a logical replication slot already enabled in the source database, there are usually jobs using that slot for real-time data acquisition—i.e., running jobs. At this point, the synchronization endpoint parses the underlying configuration or running status of the running job, extracting the set of source database objects that the logical replication slot is monitoring, capturing, and synchronizing. This set is the set of subscribed objects. This set typically contains one or more database tables, specific columns, or data ranges that are currently undergoing logical replication.

[0034] Next, the objects to be synchronized are compared with the set of subscribed objects to determine the incremental objects to be synchronized. Specifically, the objects to be synchronized configured in the data synchronization job, such as specific database tables in the Gbase8c database specified by the user, are cross-compared with the extracted set of subscribed objects. By comparing the object identifiers, such as table names and schema names, it is possible to accurately identify which objects are needed by the task and have not yet been subscribed to by the logical replication slots. These unsubscribed objects that need to be newly collected are the incremental objects to be synchronized.

[0035] For example, if the set of subscribed objects of a running job contains tables A and B, and the data synchronization job requires the synchronization of tables B and C, after comparison, table C is identified as an incremental object to be synchronized, while table B, since it has already been subscribed, is automatically removed, thus avoiding duplicate extraction and waste of resources.

[0036] Finally, the incremental objects to be synchronized can be appended to the set of subscribed objects to obtain the merge synchronization job. Specifically, in the construction and operation of real-time data warehouses, business requirements often demand extremely high continuity of data synchronization. Any shutdown or restart of the data channel will lead to interruption or high latency of the data flow. Therefore, this embodiment of the invention proposes a strategy that can automatically merge jobs of different tables under the same database without interrupting operations. That is, while ensuring that the current logical replication slot is not interrupted, not paused, and continuously capturing logs, the identified incremental objects to be synchronized are seamlessly and smoothly appended to the set of subscribed objects by dynamically refreshing the publish-subscribe mechanism or hot update job configuration of the source database. After the append is completed, the data processing logic of the new job will be automatically integrated and optimized with the logic of the original running job in the background to form a merge synchronization job sharing the same logical replication slot.

[0037] In this embodiment of the invention, by acquiring the set of subscribed objects and performing difference comparison, and then dynamically adding incremental objects to be synchronized while the logical replication slot is running continuously, not only is on-demand synchronization and deduplication merging at the fine-grained table level achieved, avoiding resource waste caused by repeated extraction, but more importantly, it ensures the continuity and stability of the existing real-time data acquisition link. It can flexibly and seamlessly adapt to the addition of new nodes and the dynamic expansion of the data synchronization architecture, breaking the deadlock of soaring operation and maintenance costs due to the addition of new tasks in traditional solutions.

[0038] Based on the above embodiments, step 110 includes: It displays a visual configuration interface that includes source configuration controls, destination configuration controls, and rule configuration controls; It receives assembly instructions triggered by manipulating the source configuration control, destination configuration control, and rule configuration control in the visual configuration interface. The assembly instructions carry connection attributes associated with the source database, object identifiers associated with the objects to be synchronized, and data processing rules. The assembly instructions are parsed and converted into code to obtain data synchronization operations.

[0039] Specifically, the process of obtaining the data synchronization job in step 110 includes: First, the synchronization endpoint displays a visual configuration interface to users or operations personnel. That is, to lower the barrier to entry for users and avoid them performing complex script writing and debugging operations directly at the database level, this embodiment of the invention adopts a drag-and-drop visual configuration method in the job generation stage. Specifically, the data synchronization task is abstracted into source, destination, preprocessing, etc., and rendered as corresponding graphical interactive controls on the front end, namely source configuration controls, destination configuration controls, and rule configuration controls.

[0040] The source configuration control allows users to input or select the source information of the data; the destination configuration control allows users to set the target database where the data will ultimately be written; and the rule configuration control is used to configure intermediate preprocessing operations or filtering logic that need to be performed during data synchronization. Users can intuitively and conveniently complete the configuration of task parameters by dragging and dropping, selecting from drop-down menus, or filling in forms on the visual configuration interface.

[0041] When a user completes the parameter settings on the front-end interface and triggers the job generation operation, such as clicking the "Submit" or "Generate Job" button, the front-end encapsulates these scattered graphical configuration parameters into a comprehensive assembly command and sends it to the back-end. This assembly command carries the necessary information for data synchronization, namely: connection attributes associated with the source database (e.g., the IP address, port, username, and password of the Gbase8c database requiring real-time synchronization); object identifiers associated with the objects to be synchronized (i.e., the specific table names, column names, or specific data ranges to be copied, flexibly selected by the user to clarify the granularity of the synchronized data); and data processing rules (i.e., the intermediate preprocessing steps such as data anonymization, format conversion, and field concatenation that need to be performed before data is transmitted from the source to the destination).

[0042] Figure 2 This is a flowchart of the overall data synchronization method provided by the present invention, as follows: Figure 2 As shown, after receiving the instruction, the backend parses and converts it into code to obtain the data synchronization job. Specifically, because the underlying logical replication slots and data acquisition plugins cannot directly recognize the graphical parameters from the frontend, the backend performs deep syntax parsing and logical mapping on the configuration information in the assembly instructions to automatically convert these configuration parameters into data processing logic and plugin code executable by the underlying environment. Through this code conversion and task encapsulation, a complete data synchronization job that can be directly triggered and invoked by the scheduling engine is finally generated.

[0043] In this embodiment of the invention, by abstracting various concepts of data synchronization and providing a drag-and-drop visual configuration interface, the background automatically parses and converts the assembly instructions into corresponding code and processing logic. The drag-and-drop method of the visual interface replaces the traditional cumbersome script and code development, avoiding the complicated operations of users at the database level. This not only significantly reduces the usage threshold and maintenance and debugging difficulty for non-technical personnel, but also improves the accuracy of job configuration and the efficiency of data channel construction.

[0044] Based on the above embodiments, in step 130, a merge synchronization job is executed to obtain the logical logs generated in the source database for the object to be synchronized through the logical replication slot. Prior to this, the following steps are also included: Send a publish command to the source database for the object to be synchronized, so as to generate a set of data change publishes associated with the object to be synchronized in the source database; By merging synchronization jobs, a logical replication slot-based connection is established between the synchronization endpoint and the data change release set, and a full data snapshot of the object to be synchronized is obtained through the connection.

[0045] Specifically, this embodiment of the invention employs a publisher / subscriber model to achieve data synchronization based on logical replication. Before formally acquiring incremental logical logs, it is necessary to complete the publishing and subscription settings and the synchronization of baseline data. This process specifically includes: First, the synchronizing end sends a publish command to the source database for the objects to be synchronized. See also Figure 2 As can be seen, in the preliminary preparation stage of real-time synchronization, it is necessary to adjust relevant configuration items for the source database, such as Gbase8c, to enable logical replication, and to publish the database tables or objects to be collected. Here, the published set is a collection of one or more objects defined by the source database (publisher). The synchronization end instructs the source database to internally generate a data change publication set associated with the objects to be synchronized by sending a publication command. This set can contain a specific table, multiple different tables, or specific data columns, defining the scope of subsequent data change tracking and distribution from the source database.

[0046] Next, by merging the synchronization job, a connection based on logical replication slots can be established between the subscriber on the synchronizing end and the data change publication set. That is, after setting up the source database publication set, the synchronizing end, as a subscriber, needs to configure logical replication subscription with the source database. This involves controlling the merge synchronization job to initiate a connection request in the background to bind the synchronizing end to the data change publication set on the source database through a reused logical replication slot.

[0047] After establishing a connection, incremental logical log retrieval does not begin immediately. Instead, a full data snapshot of the object to be synchronized is obtained through the connection. Specifically, in the initial stages of subscription establishment, the subscriber needs to receive a complete snapshot of the current state of the object to be synchronized from the publisher—a full data snapshot. This full data snapshot provides a consistent view of the data, ensuring that the subscriber has a correct and complete data baseline before subsequent incremental logical log replication begins.

[0048] In this embodiment of the invention, by introducing a table-level publish-subscribe mechanism and forcibly acquiring a full data snapshot when establishing a logical replication slot connection, flexible synchronous publishing of some or all data is achieved, avoiding unnecessary data transmission, greatly ensuring the consistency and accuracy of data synchronization, effectively solving the data conflict and inconsistency problems that are easily caused by the synchronization of multi-source data changes in a distributed environment, and providing a reliable data foundation for the construction of a demanding real-time data warehouse.

[0049] Based on the above embodiments, a release instruction for the object to be synchronized is sent to the source database to generate a data change release set associated with the object to be synchronized within the source database, including: Retrieve the target data operation type that matches the object to be synchronized; Generate an object identifier that includes the associated object of the object to be synchronized, as well as a constraint configuration instruction for the target data operation type, as a release instruction; The release command is sent to the source database so that the source database can build a set of data change releases based on change records that match the target data operation type.

[0050] Specifically, the process of generating a data change release set within the source database can include: First, it is necessary to obtain the target data operation type that matches the object to be synchronized. That is, in actual business scenarios, users' data synchronization needs for the same database table are often different. For example, for an audit log table, a user may only need to synchronize newly added data; while for a status monitoring table, a user may be more concerned with data updates or deletions. Therefore, in this embodiment of the invention, before defining the data change release set, the operation type restrictions pre-specified by the user for the object to be synchronized, i.e., the target data operation type, are obtained. Here, the target data operation type covers DML operation types, such as INSERT, UPDATE, and DELETE operations.

[0051] Subsequently, a release command is generated, containing the object identifier associated with the object to be synchronized and the constraint configuration instruction for the target data operation type. Specifically, after identifying the object to be synchronized and the specific operation type, these are encapsulated and concatenated in the background. This involves combining the object identifier (e.g., Schema name.table name) used to uniquely identify the object to be synchronized with the target data operation type to generate a constraint configuration instruction with constraints. This constraint configuration instruction serves as the final release command sent to the source database. This instruction clearly defines which table the source database should monitor and, more specifically, which change operations on that table should be monitored.

[0052] Finally, the release command can be sent to the source database, enabling it to construct a data change release set based on change records that match the target data operation type. Upon receiving the release command, the source database will create the corresponding release definition internally. During subsequent operation, when the source database captures data changes by parsing underlying logs, such as WAL logs, it will automatically filter according to the constraints in the release command, extracting only change records that belong to the specified object identifier and match the defined target data operation type, thereby constructing a precise data change release set.

[0053] In this embodiment of the invention, by obtaining the target data operation type and generating a constraint configuration instruction containing object identifier and operation type as a release instruction, fine-grained table-level and operation-level data release is achieved. This allows users to flexibly choose to copy specific DML operation types when defining release sets, enabling precise synchronization of some data. It effectively avoids the capture of irrelevant changed data and unnecessary data transmission and storage, further reducing the resource overhead of the source database and synchronization end, and greatly improving the targeting and flexibility of data synchronization, thus meeting the differentiated needs of some business scenarios.

[0054] Based on the above embodiments, step 130 includes: Determine the initial log position in the source database corresponding to the full data snapshot; Logical logs are obtained through logical replication slots; logical logs are obtained by reverse engineering the write-ahead logs generated in the source database after the initial log position.

[0055] Specifically, after the full data snapshot synchronization is completed, in order to seamlessly connect to the real-time collection of incremental data, the process of obtaining logical logs through logical replication slots may include: First, it is necessary to determine the initial log position corresponding to the full data snapshot in the source database. That is, since the source database is constantly running and being written, the full data snapshot only represents the state of the source database at the initial moment. Therefore, in order to ensure seamless integration between incremental and full data, this embodiment of the invention records the precise location identifier of the snapshot in the underlying log of the source database after obtaining the full data snapshot. This identifier is the initial log position, which serves as the boundary between full and incremental synchronization.

[0056] Then, the synchronizing end can reuse logical replication slots to retrieve write-ahead logs generated after the initial log point. These write-ahead logs, such as the WAL log, are underlying log files in the source database used to ensure transaction durability and consistency. They record all changes that occur in the source database sequentially in binary form. After the full data snapshot synchronization is complete, the synchronizing end uses logical replication slots to request and pull incremental write-ahead logs generated after the initial log point from the source database, according to time or sequence number order, thus continuously receiving the latest changed data from the source database.

[0057] Finally, because raw write-ahead logs, such as xlog, are stored in a binary structure underlying the database, external systems find them difficult to understand and use directly. Therefore, the source database can use a configured logical decoder to reverse-engineer the write-ahead log. Figure 3 This is an example diagram of the log reverse decoding process provided by the present invention. Figure 4 This is an example diagram of the logical log provided by the present invention, such as... Figure 3 and Figure 4 As shown, this operation parses the underlying change records into a row-level incremental data stream.

[0058] The data after being decrypted by the logic decoder is essentially a human-readable logic log, containing specific DML operations such as insert, update, and delete, along with their corresponding data values. The synchronization end processes these logic logs generated from the decryption sequentially, ensuring transaction consistency; therefore, this logical replication is also known in the industry as transaction replication.

[0059] In this embodiment of the invention, by locating the initial log position corresponding to the full snapshot and obtaining the write-ahead log after that position for reverse decoding, a seamless connection between the full baseline data and the incremental change data is achieved. This not only ensures zero loss and zero duplication in the data change capture process, but also guarantees extremely high data transaction consistency through sequential processing. It solves the problems of stability collapse and frequent missed or incorrect sampling in the high-concurrency write scenario of traditional CDC technology, and provides highly accurate incremental data streams for downstream businesses with stringent requirements such as real-time data warehouses.

[0060] Based on the above embodiments, the logical log is obtained by the source database reversing the write-ahead log when it detects a data change event for the object to be synchronized.

[0061] Specifically, during the operation of the logical replication mechanism, the underlying engine of the source database continuously monitors the objects to be synchronized. When DML operations such as insert, update, and delete are performed on the objects to be synchronized, these operations are identified as data change events. By monitoring data change events in real time, the instantaneous data changes in the source table can be captured.

[0062] Once a data change is detected in the source table, the parsing process for the corresponding log is immediately triggered. This involves the source database using a configured logical decoder to perform precise logical decoding (i.e., reverse decoding) of the write-ahead log (xlog). This decoding process directly transforms and extracts the table data changes from the originally obscure xlog log into a structured row data stream. After being reverse-decoded by the logical decoder, the data is essentially transformed into a logical log that has removed redundant information, is human-readable, and can be directly processed by the business system.

[0063] In this embodiment of the invention, by monitoring data change events of the object to be synchronized in real time, and using a logic decoder to perform accurate reverse decoding of the write-ahead log when a change is detected, the complex underlying binary change records are transformed into a human-readable row data stream. This not only greatly improves the accuracy and efficiency of data change identification for Gbase8c database in complex business scenarios, but also ensures that data change capture can occur almost synchronously with normal business operations, thereby providing a low-latency, highly reliable, and easy-to-process standardized incremental data stream for the entire real-time data acquisition channel.

[0064] Based on the above embodiments, step 140 includes: Based on data processing rules, the logical logs are processed to obtain a data stream in the target format; The target format data stream is routed to the destination write plugin that matches the type of the target database, and the target format data stream is written to the target database through the destination write plugin.

[0065] Specifically, the process of synchronizing the data obtained from the logical log parsing to the target database can include: First, the logical logs can be processed according to data processing rules to obtain a data stream in the target format. Specifically, since users have pre-defined the intermediate preprocessing operations required for data synchronization in the rule configuration control of the visual configuration interface, after obtaining the reverse-processed logical logs, before writing them to the target database, these configured intermediate preprocessing algorithms are automatically called in the background to perform operations such as de-identification, field filtering, format conversion, and data concatenation on the incremental data in the logical logs. After this processing, the change log data originally from the Gbase8c database is cleaned and converted into a target format data stream that conforms to the specifications and business requirements of the downstream target database.

[0066] Then, the target format data stream can be routed to a destination write plugin that matches the type of the target database, and the destination write plugin writes the target format data stream to the target database. That is, to adapt to diverse information systems and break down barriers between heterogeneous data sources, this embodiment of the invention adopts a pluggable design concept in the data writing stage. This means that the corresponding target database type is automatically matched based on the connection attributes in the job configuration, such as various relational databases, distributed message queues, or real-time data warehouse components, and a destination write plugin that highly matches it is selected. Subsequently, the target format data stream is dynamically routed to this plugin, which completes the adaptation of the underlying protocol, ultimately writing the data smoothly and seamlessly to the target database.

[0067] In this embodiment of the invention, data preprocessing is performed before writing, and a pluggable data acquisition plugin based on the logical replication characteristics of Gbase8c is introduced. This achieves deep decoupling of data acquisition, data processing, and data writing. This pluggable data synchronization architecture not only ensures that data can be written to various types of target databases in real time and stably, easily meeting the complex and diverse information system integration needs, but also realizes the separation of database read and write operations, greatly improving the performance and scalability of the entire real-time data warehouse. This allows it to flexibly and seamlessly adapt to the addition of new nodes and cluster topology adjustments, significantly reducing subsequent operation and maintenance costs.

[0068] The data synchronization device provided by the present invention is described below. The data synchronization device described below and the data synchronization method described above can be referred to in correspondence.

[0069] Figure 5 This is a schematic diagram of the data synchronization device provided by the present invention. This device is applied at the synchronization end, such as... Figure 5 As shown, the device includes: The acquisition unit 510 is used to acquire a data synchronization job to be executed, wherein the data synchronization job is configured with a source database and an object to be synchronized; The merging unit 520 is used to merge the data synchronization job with the running job associated with the logical replication slot when there is an open logical replication slot in the source database, to generate a merge synchronization job. Execution unit 530 is used to execute the merge synchronization job to obtain the logical log generated in the source database for the object to be synchronized through the logical replication slot; Synchronization unit 540 is used to synchronize the data obtained from parsing the logical log to the target database.

[0070] The data synchronization device provided by this invention acquires a data synchronization job configured with a source database and an object to be synchronized. When there is an open logical replication slot in the source database, it automatically merges the job with the running job to generate a merge synchronization job. Then, it obtains logical logs through the reused logical replication slots and synchronizes the parsed data to the target database. This solves the problems of high load caused by frequent table scans and high management overhead caused by triggers in traditional solutions. By utilizing the logical replication characteristics and replication slot reuse mechanism of the database itself, it captures change data with high real-time performance and high accuracy, and automatically optimizes the merge job in the background to avoid duplicate extraction. This effectively reduces the operating load and resource consumption of the source database, improves the flexibility and overall efficiency of data synchronization, and meets the stringent requirements of complex scenarios such as real-time data warehouses.

[0071] Based on the above embodiments, the merging unit 520 is used for: Get the set of subscribed objects corresponding to the running job; By comparing the object to be synchronized with the set of subscribed objects, an incremental object to be synchronized is obtained; While the logical replication slot is running continuously, the incremental objects to be synchronized are appended to the set of subscribed objects to obtain the merge synchronization job.

[0072] Based on the above embodiments, the acquisition unit 510 is used for: It displays a visual configuration interface that includes source configuration controls, destination configuration controls, and rule configuration controls; The system receives an assembly command triggered by operating the source configuration control, the destination configuration control, and the rule configuration control in the visual configuration interface. The assembly command carries a connection attribute associated with the source database, an object identifier associated with the object to be synchronized, and data processing rules. The assembly instructions are parsed and converted into code to obtain the data synchronization operation.

[0073] Based on the above embodiments, the device further includes a snapshot determination unit, used for: Send a release instruction to the source database for the object to be synchronized, so as to generate a data change release set associated with the object to be synchronized in the source database; Through the merge synchronization operation, a connection based on the logical replication slot is established between the synchronization end and the data change release set, and a full data snapshot of the object to be synchronized is obtained through the connection.

[0074] Based on the above embodiments, the snapshot determination unit is used for: Obtain the target data operation type that matches the object to be synchronized; Generate an object identifier that includes the object to be synchronized and a constraint configuration instruction for the target data operation type, as the release instruction; The release instruction is sent to the source database, so that the source database constructs a data change release set based on change records that conform to the target data operation type.

[0075] Based on the above embodiments, the execution unit 530 is used for: Determine the initial log position corresponding to the full data snapshot in the source database; The logical log is obtained through the logical replication slot; the logical log is obtained by reverse engineering the write-ahead log generated in the source database after the initial log position.

[0076] Based on the above embodiments, the execution unit 530 is used for: The logical log is obtained by the source database from the write-ahead log when it detects a data change event for the object to be synchronized.

[0077] Based on the above embodiments, the synchronization unit 540 is used for: Based on the data processing rules, the logical log is processed to obtain a data stream in the target format; The target format data stream is routed to a destination write plugin that matches the type of the target database, and the target format data stream is written to the target database through the destination write plugin.

[0078] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can invoke logical instructions in the memory 630 to execute a data synchronization method. This method includes: acquiring a data synchronization job to be executed, the data synchronization job being configured with a source database and an object to be synchronized; if an open logical replication slot exists in the source database, merging the data synchronization job with a running job associated with the logical replication slot to generate a merged synchronization job; executing the merged synchronization job to acquire logical logs generated in the source database for the object to be synchronized via the logical replication slot; and synchronizing the data parsed from the logical logs to the target database.

[0079] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0080] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the data synchronization method provided by the above methods, the method comprising: obtaining a data synchronization job to be executed, the data synchronization job being configured with a source database and an object to be synchronized; if an open logical replication slot exists in the source database, merging the data synchronization job with a running job associated with the logical replication slot to generate a merged synchronization job; executing the merged synchronization job to obtain a logical log generated in the source database for the object to be synchronized through the logical replication slot; and synchronizing the data parsed based on the logical log to the target database.

[0081] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the data synchronization method provided by the above methods. The method includes: acquiring a data synchronization job to be executed, the data synchronization job being configured with a source database and an object to be synchronized; if an open logical replication slot exists in the source database, merging the data synchronization job with a running job associated with the logical replication slot to generate a merged synchronization job; executing the merged synchronization job to acquire logical logs generated in the source database for the object to be synchronized through the logical replication slot; and synchronizing data parsed from the logical logs to the target database.

[0082] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0083] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data synchronization method, characterized in that, Applied to synchronization terminals, including: Obtain a data synchronization job to be executed, wherein the data synchronization job is configured with a source database and objects to be synchronized; If an open logical replication slot exists in the source database, the data synchronization job is merged with the running job associated with the logical replication slot to generate a merge synchronization job. Execute the merge and synchronization job to obtain the logical logs generated in the source database for the object to be synchronized through the logical replication slot; The data obtained from parsing the logical logs will be synchronized to the target database.

2. The data synchronization method according to claim 1, characterized in that, The step of merging the data synchronization job with the running job associated with the logical replication slot to generate a merged synchronization job includes: Get the set of subscribed objects corresponding to the running job; By comparing the object to be synchronized with the set of subscribed objects, an incremental object to be synchronized is obtained; While the logical replication slot is running continuously, the incremental objects to be synchronized are appended to the set of subscribed objects to obtain the merge synchronization job.

3. The data synchronization method according to claim 1, characterized in that, The process of acquiring the data synchronization job to be executed includes: It displays a visual configuration interface that includes source configuration controls, destination configuration controls, and rule configuration controls; The system receives an assembly command triggered by operating the source configuration control, the destination configuration control, and the rule configuration control in the visual configuration interface. The assembly command carries a connection attribute associated with the source database, an object identifier associated with the object to be synchronized, and data processing rules. The assembly instructions are parsed and converted into code to obtain the data synchronization operation.

4. The data synchronization method according to any one of claims 1 to 3, characterized in that, The step of executing the merge synchronization job to obtain the logical logs generated in the source database for the object to be synchronized through the logical replication slot also includes: Send a release instruction to the source database for the object to be synchronized, so as to generate a data change release set associated with the object to be synchronized in the source database; Through the merge synchronization operation, a connection based on the logical replication slot is established between the synchronization end and the data change release set, and a full data snapshot of the object to be synchronized is obtained through the connection.

5. The data synchronization method according to claim 4, characterized in that, Sending a release instruction to the source database for the object to be synchronized, to generate a data change release set associated with the object to be synchronized within the source database, includes: Obtain the target data operation type that matches the object to be synchronized; Generate an object identifier that includes the object to be synchronized and a constraint configuration instruction for the target data operation type, as the release instruction; The release instruction is sent to the source database, so that the source database constructs a data change release set based on change records that conform to the target data operation type.

6. The data synchronization method according to claim 4, characterized in that, The execution of the merge synchronization job, to obtain the logical logs generated in the source database for the object to be synchronized through the logical replication slot, includes: Determine the initial log position corresponding to the full data snapshot in the source database; The logical log is obtained through the logical replication slot; the logical log is obtained by reverse engineering the write-ahead log generated in the source database after the initial log position.

7. The data synchronization method according to claim 6, characterized in that, The logical log is obtained by the source database from the write-ahead log when it detects a data change event for the object to be synchronized.

8. The data synchronization method according to claim 3, characterized in that, The step of synchronizing the data obtained from the parsing of the logical log to the target database includes: Based on the data processing rules, the logical log is processed to obtain a data stream in the target format; The target format data stream is routed to a destination write plugin that matches the type of the target database, and the target format data stream is written to the target database through the destination write plugin.

9. A data synchronization device, characterized in that, Applied to synchronization terminals, including: The acquisition unit is used to acquire a data synchronization job to be executed, wherein the data synchronization job is configured with a source database and an object to be synchronized; The merging unit is used to merge the data synchronization job with the running job associated with the logical replication slot when there is an open logical replication slot in the source database, to generate a merged synchronization job. An execution unit is used to execute the merge synchronization job to obtain the logical logs generated in the source database for the object to be synchronized through the logical replication slot; The synchronization unit is used to synchronize the data obtained from parsing the logical log to the target database.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the data synchronization method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data synchronization method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data synchronization method as described in any one of claims 1 to 8.

Images