Data synchronization method and system in data analysis system

Abstract

The present disclosure relates to data synchronization in a data analysis system. A computer-implemented method includes receiving at least one log record. A type of operation performed in a source database system may be determined based on the log record. Based on the type of operation, an update strategy may be selected from a set of predefined update strategies, wherein the update strategy specifies a sequence of replication operations to be performed to apply changes described in the at least one log record to a target database system.

Description

Background Art

[0001] The present invention relates to the field of digital computer systems, and in particular to a data synchronization method in a data analysis system.

[0002] Replication is the process of maintaining a defined set of data in more than one location. It involves copying specified changes from one source location to a target location, synchronizing the data between the two locations. The source and target can be logical servers located on the same machine or on different machines in a distributed network. Several systems exist for moving data from one system to another. However, these systems may require further improvement. Summary of the Invention

[0003] The various embodiments provide methods, computer systems, and computer program products for data synchronization in a data analysis system, and describe advantageous embodiments. The embodiments of the present disclosure may be freely combined with one another, provided they are not mutually exclusive.

[0004] In one aspect, a computer-implemented method for data synchronization in a data analysis system is provided. The method includes receiving at least one log record describing an operation performed in a source database system of the data analysis system; determining a type of operation performed in the source database system based on the log record; and selecting an update strategy from a set of predefined update strategies based on the type of operation, the update strategy specifying a sequence of replication operations to be performed to apply changes described in the at least one log record to a target database system of the data analysis system. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In the following examples, embodiments of the invention are explained in more detail, by way of example only, with reference to the accompanying drawings, in which:

[0006] Figure 1 is a block diagram of a data processing system according to an example of the present subject matter.

[0007] Figure 2 is a flowchart of a method for enabling changes made in a source database system to be applied to a target database system according to examples of the present subject matter.

[0008] Figure 3 is a flowchart of a method for applying changes made in a source database system to a target database system according to examples of the present subject matter.

[0009] Figure 4 is a flowchart of a method for applying changes made in a source database system to a target database system according to examples of the present subject matter.

[0010] Figure 5 One embodiment of an example computer system is shown. DETAILED DESCRIPTION

[0011] The description of various embodiments of the present invention is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, practical applications, or technical improvements over technologies found in the marketplace, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0012] The present subject matter enables optimal use of multiple update strategies. This can conserve processing resources, as the update strategy selection can be based on resource utilization performance. Multiple update strategies can be provided depending on the context of the data analytics system. For example, for performance reasons or because the software's code maintenance is due, the data analytics system's incremental update replication software / technology may need to be upgraded to a completely new technology. This can result in multiple update strategies, namely, a first incremental update replication software and a second incremental update replication software that is an upgrade of the first incremental update replication software. However, the first technology may be mature and well-tested, while the new technology may have limitations in the supported data changes that can be replicated. The present subject matter ensures robustness while incrementally applying technology transitions within a continuous delivery model by selecting between the replacement and new technologies. Furthermore, runtime-specific errors may occur related to the customer's specific system configuration. By selecting either the first or second technology stack, such situations (which would cause downtime in the replication setup and require a manual maintenance window for correction) can be smoothly handled by the replication system itself, without requiring knowledge of version-specific limitations or manual intervention during replication system setup.

[0013] The data analysis system may be, for example, a data warehousing system or a master data management system. The data analysis system may enable data warehousing or master data management, or another technique that utilizes source and target database systems, wherein the target database system includes a target database configured to receive / include a copy of the contents of a corresponding source database of the source database system. The source database system may be connected to the target database system via a connection. For example, the connection may be a TCP / IP connection or another connection that enables data communication between the source and target database systems. The source database system may be, for example, a transaction engine, and the target database system may be an analytics engine. For example, the source database system may be an online transaction processing (OLTP) system, and the target database system may be an online analytical processing (OLAP) system. The source database system may include a source dataset, and the target database system may include a target dataset. The source dataset may be part of a source database, and the target dataset may be part of a target database. The source and target datasets may be stored in the same or different formats. The formats may differ in terms of encryption, compression, row-oriented versus column-oriented storage, and so on. For example, the source dataset may be stored in a row-oriented format, while the target dataset may be stored in a column-oriented format. In other words, the target dataset may be stored in columns rather than rows.The contents of the source dataset may be changed by one or more database transactions.

[0014] A database transaction refers to a unit of work performed against a database that is processed independently of other transactions in a consistent and reliable manner. A database transaction generally represents any change within a database. For example, a database transaction can be issued to a database system using a language like SQL wrapped in a transaction, using a pattern similar to the following: s1) begin transaction, s2) perform a set of data operations and / or queries, s3) commit the transaction and end it if no errors occur, s4) rollback the transaction and end it if an error occurs. The transaction commit operation applies all data manipulations within the scope of the transaction and persists the results to the database. The processing steps of a database transaction can be steps s1) and s2) of the pattern, while the apply step can be the commit of the transaction to end it.

[0015] A database may have one or more tablespaces and tables with associated indexes. A tablespace is a set of volumes on disk that holds the actual data sets that store tables. Each table may be stored in a tablespace. Data operations may be database operations, that is, operations on tables, tablespaces, or indexes. These operations may have different categories depending on the type of data being operated on. For example, operations performed at the tablespace level may be in the first category, while operations performed on tables may be in the second category. Furthermore, each operation category may include one or more types of operations. For example, the second category of operations may include first-type operations such as insert, delete, and update operations on a table, and second-type operations that change the table's structure. Depending on the type of operation, different tools may be used to perform the operation. For example, data operations may be performed using SQL statements or utilities (such as DB2 utilities). Utilities may be used for operations in the first category, while SQL statements may be used for operations in the second category. Utilities can perform large-scale operations across one or more tablespaces. For example, a utility may load thousands or even millions of rows into a database or reorganize data in a tablespace. SQL statements may be used to perform routine operations such as insert, update, and delete operations. In addition, SQL statements may be used to change the structure of a table. The type of operation may be, for example, REGULAR indicating the normal flow of insert, update, and delete operations, STRUCTURAL_CHANGE indicating a structural change, UTILITY_ACTION_IN_Execute indicating a utility action in progress, or UTILITY_ACTION_DONE indicating completion of a utility action on a table.

[0016] Therefore, using the same update strategy for different types of operations may not be optimal, as an update strategy may only be executed for certain types of operations and not for others. The present subject matter addresses this issue by selecting an appropriate update strategy based on the type of operation. The update strategy may, for example, be one of the predefined update strategies. The predefined update strategies may, for example, include a first update strategy that uses a batch load and a second update strategy that uses an incremental load. Thus, the first update strategy may be designated as a batch load-based update strategy, and the second update strategy may be designated as an incremental load-based update strategy. Each of the first and second update strategies may be implemented in different ways, resulting in different update strategies. An incremental load-based update strategy may implement incremental loads of changes. An incremental load-based update strategy may only update data that has changed since the previous update. Because fewer data rows may be copied during each update, incremental copying may be more efficient than batch copying. A batch load-based strategy may load the entire table data or a group of table partitions at a given point in time. In one implementation example, an incremental load-based update strategy may copy data changes recorded from INSERT, UPDATE, and DELETE statements on a table. In another implementation example, an incremental load-based update strategy may replicate logged data changes from INSERT, UPDATE, DELETE, and ALTER statements on a table.

[0017] A log record is log information that describes the changes caused by one or more operations of a database transaction performed on one or more data records of a source table in a database. For example, log information is provided so that it can be used by a target database system to apply the same changes to a target table corresponding to the source table. For example, if the source and target tables have identical contents before the changes are applied to the source table, log information can be provided so that the same changes can be applied to the target table, resulting in the target table having the same contents as the source table modified by the changes. Log information may include, for example, general log attributes and other attributes depending on the type of change associated with the log record. General log attributes may include, for example, a log record sequence number (LRSN), which is a unique ID for the log record indicating the logical order of the changes; a timestamp indicating the time the log record was created; and a transaction ID number, which is a reference to the database transaction that generated the log record. Other attributes may include, for example, a page ID indicating the page ID of the modified page, the length and offset of the page, and before and after images containing the values ​​of the bytes of the page before and after the page change, for an update change type. The page may include one or more data records of the database. The before and after images may include the values ​​of one or more data records before and after the change is applied, respectively. Each data record referenced by the log record may include multiple attributes. The data record may be uniquely identified, for example, via an ID attribute, a set of key attributes in a corresponding database table, or the like.

[0018] According to one embodiment, at least one log record includes a sequence of log records for a specific table of the source database system, wherein the operation type is determined based on the sequence. For example, if the operation sequence represents a normal flow of insert, update, and delete operations, the operation type may indicate a normal change operation performed on the table.

[0019] According to one embodiment, the method includes applying changes described in at least one log record according to a selected update strategy (primary update strategy). This subject matter can enable a hybrid database change replication system that uses multiple replication strategies to synchronize data between a source system and a target system. This can make the same data available in different systems. By making data available on multiple systems, this can facilitate large-scale data sharing between systems and distribute network load across multiple systems.

[0020] According to one embodiment, the method includes detecting a failure to apply a change. In response to the detection, a secondary update strategy can be selected, and the change can be applied based on the secondary update strategy. For example, the secondary update strategy can be a strategy that uses batch loads. This embodiment can save processing resources that would otherwise be required to first fix the problem with the primary update strategy and then use it again after the fix is ​​applied.

[0021] In one example, an update policy such as the primary update policy may be used as a default update policy for the data analysis system. For example, the method may include that after applying changes according to the secondary update policy, a switch back to the default update policy may be performed.

[0022] According to one embodiment, the type of operation includes at least one of a regular change for adding, deleting, or updating records of a table, a structural change for changing the structure of a table, or an activity notification related to a utility action performed on a source database system. The utility action may be, for example, a DB2 utility action, such as the following DB2 operations: LOAD, REORG TABLESPACE, DISCARD FROM TABLE, etc. Logging provides a means of detecting such DB2 utility actions.

[0023] According to one embodiment, the update strategy includes at least one of the following: a regular record add, delete, or update operation; a batch load of data from the target database system to the source database system; or a suspend operation for suspending modifications to a table in the target database system. For example, if a utility action is executing, suspending modifications to a specific table can be performed. Once the utility action is complete, an update strategy based on the batch load can be selected and applied.

[0024] Figure 1 is a block diagram of a data processing system (or data analysis system) 100 according to an example of the present subject matter. Data processing system 100 may be configured to synchronize data between a source database system 101 and a target database system 103 using a data synchronization system 102 according to an example of the present subject matter. Source database system 101 may be, for example, an online transaction processing (OLTP) system. Target database system 103 may be, for example, an online analytical processing (OLAP) system. Communication between source database system 101 and target database system 103 may be performed, for example, via a TCP / IP communication layer.

[0025] Source database system 101 includes one or more source tables 105 of source database 110 and a transaction recovery log 106. Source tables 105 can be relational tables in DB2® for z / OS®, DB2 for Linux, UNIX, and Windows, and Oracle. Entries or log records of transaction recovery log 106 describe changes to rows of source tables 105 at source database system 101. Figure 1Example contents of log record 130 are shown. Log record 130 may include a timestamp, LRSN, and attribute changes. More specifically, a log record in transaction recovery log 106 may, for example, contain information defining (1) the table being changed, (2) the values ​​of the key columns in the row being changed, (3) the old and new values ​​of all columns of the row being changed, and (4) the transaction (unit of work) that caused the change. By definition, an insert is a new data record and therefore has no old values. For a delete change, by definition there are no new data records, only old data records. Thus, a transaction log record for an inserted row may contain only the new column values, while a transaction log record for a deleted row may contain only the old column values. A transaction log record for an updated row may contain both the new and old values ​​of all row columns. The order of log records in the transaction recovery log may reflect the order of the transaction's change operations, and the order of transaction commit records may reflect the order in which the transactions were completed. The type of row operation in a transaction log record may, for example, be delete, insert, or update.

[0026] The data synchronization system 102 includes a log reader 104. The log reader 104 can read log records of the transaction recovery log 106 and provide them to a change record classifier 120. The change record classifier 120 can classify the log records based on the changes in the log records, so that the selection module 121 can select an update strategy based on the classification of the log records and perform updates based on the selected update strategy. Update strategy selection and application of changes using the selected strategy can be performed periodically, for example, every hour, or can be performed automatically when a predefined number of log records (for example, 1,000 log records) is saved in the transaction recovery log 106. In each iteration, only newly added log records relative to previously provided log records can be processed. The selected update strategy can, for example, include a batch load-based update strategy or one or more incremental update strategies. Synchronization can be performed differently for the batch load-based update strategy and the incremental update strategy.

[0027] Updates using any of the selected incremental update strategies can be performed using the corresponding application algorithm as follows. The log reader 104 can be configured to perform log shipping of the transaction recovery log 106 to the target database system 103 based on the incremental update strategy selected by the selection module 121. For example, shipping can be performed by sending a log record stream formed by log records of the transaction recovery log 106. The log stream being shipped can, for example, be associated with a stream ID. The stream of log records can, for example, be a stream of merged log records. This can enable efficient processing of log records at the target database system. The target database system 103 includes a plurality of applications 108.1 and 108.2, each associated with a corresponding incremental update strategy. Figure 1For illustrative purposes, only two apply algorithms are shown, but the present invention is not limited to two apply algorithms. Target database system 103 further includes one or more target table replicas 113 of target database 111. Target database system includes a log stream interface for receiving a log stream from source database system 101. Each of applications 108.1 and 108.2 can be configured to receive a stream of log records via the log streaming interface. Each of applications 108.1 and 108.2 can buffer received log records and merge changes into batches to improve efficiency when applying modifications to tables 113 of target database 111, for example, via a batch load interface.

[0028] In another example, a batch load (snapshot update) can be performed between source database system 101 and target database system 103 based on a selected batch-based update strategy. This load can be a load of all table data or a set of table partitions at a given point in time, and is performed directly from the source database system to the target database system. The data on the target database system can reflect the state of the source database system at the time the load is performed.

[0029] Although shown as separate components, in another example, the data synchronization system 102 can be part of the source database system 101. In one example, the source database system 101 and the target database system 103 can be on the same system or on different systems in a distributed network. In addition, the data synchronization system 102, the source database system 101, and / or the target database system 103 can be implemented as a computer system, as described below with respect to Figure 5 System 600 is described.

[0030] Figure 2 is a flow chart of a method for applying changes made in a source database system to a target database system according to an example of the present subject matter. For purposes of explanation, Figure 2 The method described in Figure 1 The system is implemented in the system shown in, but is not limited to this implementation method. Figure 2 The method may be performed, for example, by the data synchronization system 102 .

[0031] At block 201, at least one log record may be received. The log record describes at least one operation performed on one or more source tables of the source database system 101. The at least one log record may be a record of the transaction recovery log 106. For example, the log reader 104 may read or receive the at least one log record from the source database system 101. In one example, the at least one log record includes a sequence of log records for a particular table 105 of the source database system 101. The at least one log record may be automatically received, for example, when the at least one log record is created in the transaction recovery log 106. In another example, the at least one log record may be received in response to a request sent to the source database system 101.

[0032] At block 203, a type of operation performed in the source database system 101 can be determined based on the at least one log record. The type of operation can be determined, for example, using a change record classifier 120. For example, after reading or receiving the at least one log record from the transaction recovery log 106 of the source database system, the change record classifier 120 can be used to decode and classify the change type. The classification of the change type indicates the type of operation. The type of operation can include, for example, a regular change to add, delete, or update a record of a table, a structural change to change the structure of a table, or an activity notification related to a utility action performed on the source database system. For example, the change record classifier 120 can output several types of operations, such as a regular attribute value change in the source table 105, a structural change to copy a source table, or a utility call to modify the source table 105.

[0033] Based on the type of operation, an update strategy may be selected from a set of predefined update strategies at block 205. The update strategy specifies a sequence of replication operations to be performed in order to apply the changes described in the at least one log record to the target database system 103. For example, a mapping between the operation type and the update strategy may be provided. The mapping may be used to automatically select the update strategy.

[0034] Figure 3 is a flow chart of a method for applying changes made in a source database system to a target database system according to an example of the present subject matter. For purposes of explanation, Figure 3 The method described in Figure 1 The system is implemented in the system shown in, but is not limited to this implementation method.

[0035] The update strategy selected in step 205 may be applied at block 301 to at least one log record (e.g., Figure 2(processed in the method of ). At block 303, a failure in the application of the change can be detected. For example, data synchronization system 102 can observe the application of data changes to the target database system by any implemented update strategy and can detect unexpected error conditions that prevent the application of the replicated data changes by the selected data update strategy. In response to the detection, a secondary update strategy can be selected at block 305, and the change can be applied based on the secondary update strategy at block 307. Selecting another update strategy as a fallback implementation enables automatic recovery from these runtime errors without downtime or user interaction. For example, if an incremental update strategy fails to apply a log record classified as a REGULAR operation, the failed log record is referenced to trigger selection module 121. Selection module 121 then selects a different implementation technology (e.g., and the corresponding application algorithm 108.1 or 108.2), loads the implementation, and forwards the log record to the corresponding implementation. Selection module 121 can be configured to forward log records at transaction boundaries (e.g., at the start of a transaction). In this way, overall consistency can be guaranteed by rolling back failed transactions and replaying them using different technologies. When a transaction fails via the first strategy, the operations performed so far may have to be undone, and the secondary strategy can be used to retry all operations.

[0036] Figure 4 is a flow chart of a method for applying changes made in a source database system to a target database system according to an example of the present subject matter. For purposes of explanation, Figure 4 The method described in Figure 1 The system is implemented in the system shown in, but is not limited to this implementation. Figure 4 The method may be performed, for example, by the data synchronization system 102 .

[0037] At block 401, the change record classifier 120 can determine which of the predefined classes an incoming change belongs to based on the sequence of one or more log records for the table. For example, the following classes can be defined: a REGULAR class indicating the normal flow of insert, update, and delete operations; a STRUCTURAL_CHANGE class indicating a structural change (e.g., if the change was caused by an ALTERTABLE statement and the change must be committed); a UTILITY_ACTION_IN_EXECUTION class indicating a utility action is in progress; and a UTILITY_ACTION_DONE class indicating that a utility action on the table is complete. Classification can be performed based on the parsing results / decomposition of a particular log record.

[0038] At block 403, the change category is used by selection module 121 to dynamically switch between available data replication strategies. For example, for each table, selection module 121 may select an update strategy based on the class determined in block 401. If the class is REGULAR, the actual selection may be based on performance metrics. For example, if the number of regular operations exceeds a threshold, a batch load may be selected; otherwise, an incremental load may be used. In the case of a STRUCTURAL_CHANGE classification, a batch load is immediately triggered. If UTILITY_ACTION_IN_EXECUTE is seen, the table is temporarily suspended from the synchronization process. Thus, changes are no longer actively applied. Then, when UTILITY_ACTION_DONE is seen, a batch load is triggered. Thus, in cases where a change record encodes changes that may not be incrementally applied from source table 105 to target table 113, selection module 121 may select a batch load strategy to synchronize a snapshot of the source table. After synchronizing the table with the batch load strategy, the system can use the timestamp of the last committed transaction in the previously synchronized table snapshot to switch back to an update strategy based on incremental loads to resume processing.

[0039] Figure 5 Indicates that it is suitable for realizing the above Figure 2 、 Figure 3 and / or Figure 4 The computer system 600 may implement at least one or more of the functions of the described methods.

[0040] It should be understood that the methods described herein are at least partially non-interactive and are performed automatically by a computerized system, such as a server or embedded system. However, in exemplary embodiments, the methods described herein may be implemented in a (partially) interactive system. These methods may further be implemented in software 612, 622 (including firmware 622), hardware (processor) 605, or a combination thereof. In exemplary embodiments, the methods described herein are implemented in software as an executable program and executed by a special-purpose or general-purpose digital computer (e.g., a personal computer, workstation, minicomputer, or mainframe computer). System 600 includes a general-purpose computer 601.

[0041] In an exemplary embodiment, in terms of hardware architecture, Figure 5As shown, computer 601 includes a processor 605, a memory (main memory) 610 coupled to a memory controller 615, and one or more input and / or output (I / O) devices (or peripheral devices) 10, 645 communicatively coupled via a local input / output controller 635. As is known in the art, the input / output controller 635 can be, but is not limited to, one or more buses or other wired or wireless connections. The input / output controller 635 may have additional elements omitted for simplicity, such as controllers, buffers (cache memories), drivers, repeaters, and receivers to enable communication. Further, the local interface may include address, control, and / or data connections to enable appropriate communication between the aforementioned components. As described herein, the I / O devices 10, 645 may generally include any common cryptographic card or smart card known in the art.

[0042] Processor 605 is a hardware device for executing software, particularly software stored in memory 610. Processor 605 can be any custom or commercially available processor, a central processing unit (CPU), a secondary processor among several processors associated with computer 601, a semiconductor-based microprocessor (in the form of a microchip or chipset), or generally any device for executing software instructions.

[0043] The memory 610 may include any one or a combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.) and non-volatile memory elements (e.g., ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM)). Note that the memory 610 may have a distributed architecture in which different components are remote from each other but can be accessed by the processor 605.

[0044] The software in the memory 610 may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions, particularly those involved in the embodiments described herein. Figure 5 In the example of , the software in memory 610 includes instructions 612, such as instructions for managing a database, such as a database management system.

[0045] The software in memory 610 also typically includes a suitable operating system (OS) 611. The OS 611 essentially controls the execution of other computer programs, such as possibly software 612 for implementing the methods described herein.

[0046] The methods described herein may be in the form of a source program 612, an executable program 612 (object code), a script, or any other entity comprising a set of instructions to be executed 612. When in the form of a source program, the program may need to be translated via a compiler, assembler, interpreter, etc., which may or may not be included in the memory 610, in order to operate properly in conjunction with the OS 611. Furthermore, the methods may be written in an object-oriented programming language with classes for data and methods, or in a procedural programming language with routines, subroutines, and / or functions.

[0047] In an exemplary embodiment, a conventional keyboard 650 and mouse 655 may be coupled to input / output controller 635. Other output devices such as I / O devices 645 may include input devices such as, but not limited to, printers, scanners, microphones, and the like. Finally, I / O devices 10, 645 may further include devices that transmit both input and output, such as, but not limited to, network interface cards (NICs) or modulators / demodulators (for accessing other files, devices, systems, or networks), radio frequency (RF) or other transceivers, telephone interfaces, bridges, routers, and the like. I / O devices 10, 645 may be any common cryptographic card or smart card known in the art. System 600 may further include a display controller 625 coupled to display 630. In an exemplary embodiment, system 600 may further include a network interface for coupling to network 665. Network 665 may be an IP-based network for communicating between computer 601 and any external servers, clients, and the like via a broadband connection. Network 665 transmits and receives data between computer 601 and external systems 30, which may be related to performing some or all of the functionality of the methods discussed herein. In an exemplary embodiment, network 665 may be a managed IP network managed by a service provider. Network 665 may be implemented wirelessly, for example, using wireless protocols and technologies (e.g., WiFi, WiMax, etc.). Network 665 may also be a packet-switched network, such as a local area network, wide area network, metropolitan area network, internet network, or other similar type of network environment. Network 665 may be a fixed wireless network, a wireless local area network (LAN), a wireless wide area network (WWAN), a personal area network (PAN), a virtual private network (VPN), an intranet, or other suitable network system, and may include equipment for receiving and transmitting signals.

[0048] If computer 601 is a PC, workstation, smart device, etc., the software in memory 610 may also include a basic input / output system (BIOS) 622. The BIOS is a collection of basic software routines that initialize and test hardware at startup, start OS 611, and support data transfer between hardware devices. The BIOS is stored in ROM so that it can be executed when computer 601 is activated.

[0049] When the computer 601 is in operation, the processor 605 is configured to execute software 612 stored in the memory 610, transfer data to and from the memory 610, and generally control the operation of the computer 601 according to the software. The methods and OS 611 described herein are read in whole or in part by the processor 605, possibly cached within the processor 605, and then executed.

[0050] When the systems and methods described herein are implemented in software 612, such as Figure 5 As shown in , the method can be stored on any computer-readable medium (such as storage device 620) for use by or in conjunction with any computer-related system or method. Storage device 620 may include disk storage, such as HDD storage.

[0051] This topic may include the following terms

[0052] Clause 1: A computer-implemented method for data synchronization in a data analysis system, the method comprising:

[0053] receiving at least one log record, the log record describing an operation performed in a source database system of the data analysis system;

[0054] determining a type of operation performed in the source database system based on the log record; and

[0055] Based on the operation type, an update strategy is selected from a set of predefined update strategies, the update strategy specifying a sequence of replication operations to be performed in order to apply the changes described in the at least one log record to a target database system of the data analysis system.

[0056] Clause 2: The method of clause 1, wherein the at least one log record comprises a sequence of log records for a specific table of the source database system, the operation type being determined based on the sequence.

[0057] Clause 3: The method of clause 1 or 2, wherein determining the type of operation comprises comparing a number of operations indicated in the at least one log record to a predefined threshold, and determining the type of operation based on the comparison.

[0058] Clause 4: The method of any of the preceding clauses 1-3, wherein the method comprises applying changes described in the at least one log record according to a selected update policy.

[0059] Clause 5: The method of clause 4, wherein the method comprises detecting a failure to apply the change; responsive to the detecting, selecting a secondary update policy; and applying the change based on the secondary update policy.

[0060] Clause 6: The method of any of the preceding clauses 1-5, wherein the operation type comprises at least one of a regular change for adding, deleting, or updating records of a table, a structural change for changing the structure of the table, or a utility action performed on the source database system.

[0061] Clause 7: A method according to any of the preceding clauses 1-6, wherein the update strategy uses at least one of a regular record add, delete or update operation, a batch load of the target database system with data originating from the source database system, or a suspend operation for suspending modifications to a specific table of the target database system.

[0062] The present invention may be a system, method, and / or computer program product at any possible level of integrated technical detail. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform aspects of the present invention.

[0063] A computer-readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punched card) or a raised structure in a groove having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted by a wire.

[0064] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0065] The computer-readable program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, as well as procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits (including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs)) may execute the computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuits to perform aspects of the present invention.

[0066] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0067] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine such that the instructions are executed via the processor of the computer or other programmable data processing apparatus to create a device for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to function in a specific manner such that the computer-readable storage medium having the instructions stored therein comprises an article of manufacture including instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0068] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other apparatus that causes a series of operating steps to be performed on a computer, other programmable apparatus or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus or other device implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0069] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to different embodiments of the present invention. To this end, each box in the flowchart or block diagram can represent a part of a module, segment or instruction, which includes one or more executable instructions for realizing the specified logical function. In some alternative embodiments, the functions marked in the box may not occur in the order marked in the figure. For example, depending on the functions involved, the two boxes shown in succession can actually be completed as a step, executed simultaneously, substantially simultaneously, in a manner overlapping in part or all of time, or these boxes can sometimes be executed in the opposite order. It will also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented by a system based on special-purpose hardware, which performs a specified function or action or performs a combination of special-purpose hardware and computer instructions.

Claims

1. A computer-implemented method for data synchronization in a data analysis system, the method comprising: receiving at least one log record, the log record describing an operation performed in a source database system of the data analysis system; determining a type of operation performed in the source database system based on the log record; and selecting, based on the type of the operation, an update strategy from a set of predefined update strategies, the update strategy specifying a sequence of replication operations to be performed in order to apply the changes described in the at least one log record to a target database system of the data analysis system, The update strategy uses at least one of: a regular record addition, deletion, or update operation, a batch load of the target database system with data originating from the source database system, or a suspend operation for suspending modification of a specific table of the target database system. 2 . The method according to claim 1 , wherein the at least one log record comprises a sequence of log records about a specific table of the source database system, and the type of the operation is determined based on the sequence.

3. The method according to claim 1, wherein Determining the type of the operation includes comparing the number of operations indicated in the at least one log record with a predefined threshold, and determining the type of the operation based on a result of the comparison.

4. The method according to claim 1, wherein The method includes applying changes described in the at least one log record according to the selected update policy.

5. The method according to claim 4, wherein The method comprises: detecting a failure to apply said changes; In response to the detecting, selecting a secondary update strategy; and The changes are applied based on the secondary update policy.

6. The method of claim 1, wherein the type of operation comprises at least one of: a regular change for adding, deleting, or updating records of a table, a structural change for changing a structure of the table, or a utility action performed on the source database system.

7. A computer program product having computer readable program code embodied therewith, the computer readable program code being configured to cause a processor to: receiving at least one log record, the log record describing an operation performed in a source database system of the data analysis system; determining a type of operation performed in the source database system based on the log record; and selecting, based on the type of the operation, an update strategy from a set of predefined update strategies, the update strategy specifying a sequence of replication operations to be performed in order to apply the changes described in the at least one log record to a target database system of the data analysis system, The update strategy uses at least one of: a regular record addition, deletion, or update operation, a batch load of the target database system with data originating from the source database system, or a suspend operation for suspending modification of a specific table of the target database system. 8 . The computer program product of claim 7 , wherein the at least one log record comprises a sequence of log records regarding a specific table of the source database system, the type of the operation being determined based on the sequence.

9. The computer program product of claim 7, wherein: The computer readable program code is configured to cause the processor to determine the type of the operation by comparing a number of operations indicated in the at least one log record with a predefined threshold and determining the type of the operation based on a result of the comparison.

10. The computer program product of claim 7, wherein the computer readable program code is further configured to cause the processor to apply changes described in the at least one log record according to the selected update policy.

11. The computer program product of claim 10, wherein the computer readable program code is further configured to cause the processor to: detecting a failure to apply said changes; In response to the detecting, selecting a secondary update strategy; and The changes are applied based on the secondary update policy.

12. The computer program product of claim 7, wherein: The type of operation includes at least one of: a regular change for adding, deleting, or updating records of a table, a structural change for changing a structure of the table, or a utility action performed on the source database system.

13. A computer system for data synchronization in a data analysis system, comprising: a memory configured to store executable instructions; as well as a processor communicatively coupled to the memory and configured to execute the executable instructions to: receiving at least one log record, the log record describing an operation performed in a source database system; determining a type of operation performed in the source database system based on the log record; and selecting, based on the type of the operation, an update strategy from a set of predefined update strategies, the update strategy specifying a sequence of replication operations to be performed in order to apply the changes described in the at least one log record to a target database system, The update strategy uses at least one of: a regular record addition, deletion, or update operation, a batch load of the target database system with data originating from the source database system, or a suspend operation for suspending modification of a specific table of the target database system. 14 . The computer system of claim 13 , wherein the at least one log record comprises a sequence of log records about a specific table of the source database system, the type of the operation being determined based on the sequence.

15. The computer system according to claim 13, wherein: The processor is configured to determine the type of the operation by comparing a number of operations indicated in the at least one log record with a predefined threshold and determining the type of the operation based on a result of the comparison.

16. The computer system according to claim 13, wherein: The processor is configured to apply changes described in the at least one log record according to the selected update policy.

17. The computer system according to claim 16, wherein: The processor is further configured to: detecting a failure to apply said changes; In response to the detecting, selecting a secondary update strategy; and The changes are applied based on the secondary update policy.

18. The computer system according to claim 13, wherein: The type of operation includes at least one of: a regular change for adding, deleting, or updating records of a table, a structural change for changing a structure of the table, or a utility action performed on the source database system.

Images