Data processing method and device, storage medium and electronic equipment

Abstract

The invention relates to a data processing method and device, a storage medium and electronic equipment. At least one archiving log corresponding to a first database can be obtained, and the archiving log is used for recording data change operation of the first database; generating a plurality of preset log files, wherein at least one archiving log is stored in each preset log file; for each preset log file, obtaining a transaction identifier of a data operation transaction corresponding to each archiving log in the preset log file, the data operation transaction being a logic working unit of a group of data change operations; performing multi-thread parallel scanning analysis on the plurality of preset log files to obtain an analysis result of each archived log; and according to the transaction identifier and the analysis result of each archiving log, each data operation transaction is sent to a target terminal, the target terminal is used for executing a data change operation on a second database according to the data operation transaction, and the second database is a database performing data synchronization processing with the first database.

Description

Technical Field

[0001] The present disclosure relates to the field of database technology, and in particular, to a data processing method, device, storage medium and electronic device. Background Art

[0002] With the continuous growth of data volume and the continuous changes in business needs, enterprises have an increasingly urgent need for real-time data synchronization and change data capture in database systems. In Oracle database systems, incremental change data capture is a crucial link in database synchronization and data backup. In related technologies, archive logs are mainly used to record every data change operation that occurs in the database during database operation, including insertion, update, deletion, etc. During database replication and synchronization, archive logs can help ensure data consistency between different database instances.

[0003] The traditional way of processing captured archive logs is inefficient when processing large-scale archive data, causing serious data synchronization delay problems. Summary of the invention

[0004] The purpose of the present disclosure is to provide a data processing method, device, storage medium and electronic device.

[0005] In order to achieve the above object, the first aspect of the present disclosure provides a data processing method, the method comprising: Obtain at least one archive log corresponding to the first database, where the archive log is used to record data change operations of the first database; Generate a plurality of preset log files, each of which stores at least one archived log; For each of the preset log files, a transaction identifier of a data operation transaction corresponding to each of the archive logs in the preset log file is obtained, wherein the data operation transaction is a logical work unit of a group of data change operations; wherein, for each of the data operation transactions, the data operation transaction corresponds to multiple archive logs, and the multiple archive logs corresponding to the data operation transaction are from the same or different preset log files; Performing multi-threaded parallel scanning and parsing on the plurality of preset log files to obtain the parsing result of each archived log; According to the transaction identifier and parsing result of each archive log, each data operation transaction is sent to a target terminal respectively. The target terminal is used to perform the data change operation on a second database according to the data operation transaction. The second database is a database for data synchronization with the first database.

[0006] Optionally, the parsing result includes a log type of each archived log; and the performing multi-threaded parallel scanning and parsing on the plurality of preset log files includes: Starting a preset number of log parsers, wherein the preset number of log parsers corresponds one-to-one to the preset number of preset log files; For each preset log file in the preset number of preset log files, scanning each archived log in the preset log file line by line by using a target log parser corresponding to the preset log file; For the first archive log currently scanned, after parsing the first archive log by the target log parser, the log type of the first archive log is obtained; The sending each of the data operation transactions to the target terminal according to the transaction identifier and the parsing result of each of the archive logs comprises: For the first archive log currently scanned in each of the archive logs, according to the transaction identifier and log type of the first archive log, the data change operation corresponding to the first archive log is stored in the target cache space corresponding to the target transaction identifier; wherein the target transaction identifier is the transaction identifier of the data operation transaction corresponding to the first archive log; In a case where all data change operations under the data operation transaction corresponding to the target transaction identifier are stored in the target cache space, the data operation transaction corresponding to the target cache space is sent to the target terminal.

[0007] Optionally, the log type includes a transaction commit log, and the transaction commit log is used to indicate the end of a corresponding data operation transaction; The sending of the data operation transaction corresponding to the target cache space to the target terminal comprises: In a case where the log type of the first archive log is the transaction commit log, determining whether the remaining preset log files that have not been scanned completely except the preset log file include the second archive log corresponding to the target transaction identifier; When it is determined that the second archive log is included in the remaining preset log files that have not been scanned except the preset log file, after storing the data change operation corresponding to the second archive log in the target cache space corresponding to the target transaction identifier, the data operation transaction corresponding to the first archive log is sent to the target terminal; When it is determined that the second archive log is not included in the remaining preset log files that have not been scanned except the preset log file, sending the data operation transaction corresponding to the first archive log to the target terminal; The second archive log is an archive log corresponding to the target transaction identifier and other than the first archive log that has not been scanned completely.

[0008] Optionally, the log type further includes a transaction start log or a data modification log, wherein the transaction start log indicates the start of a corresponding data operation transaction, and the data modification log indicates the data change operation of the corresponding data operation transaction; The storing, for each of the archive logs, a first archive log currently scanned, according to a transaction identifier and a log type of the first archive log, a data change operation corresponding to the first archive log to a target cache space corresponding to a target transaction identifier includes: Determine whether a target cache space corresponding to the target transaction identifier has been created in a preset cache space; In the case where the target cache space is not created in the preset cache space, if the log type of the first archive log is the transaction start log, the target cache space is created in the preset cache space; if the log type of the first archive log is the data modification log, after the target cache space is created in the preset cache space, the data change operation recorded in the first archive log is stored in the target cache space; if the log type of the first archive log is the transaction commit log, the target cache space is created in the preset cache space; In the case that the target cache space has been created in the preset cache space, if the log type of the first archive log is the data modification log, the data change operation recorded in the first archive log is stored in the target cache space; if the log type of the first archive log is the transaction commit log, it is determined whether all data change operations under the data operation transaction corresponding to the target transaction identifier are stored in the target cache space, and if it is determined that all data change operations under the data operation transaction corresponding to the target transaction identifier are stored in the target cache space, the data operation transaction corresponding to the target cache space is sent to the target terminal.

[0009] Optionally, each of the preset log files is configured with a corresponding Bloom filter, and the Bloom filter is used to characterize a set of transaction identifiers of archived logs in the corresponding preset log file; the method further includes: For each of the preset log files, recording the transaction identifier of the data operation transaction corresponding to each of the archive logs in the preset log file to the Bloom filter corresponding to the preset log file; In the case where the log type of the first archive log is the transaction commit log, determining whether the remaining preset log files that have not been scanned completely except the preset log file contain the second archive log corresponding to the target transaction identifier includes: In the case where it is determined that the target transaction identifier is located in the target Bloom filter, determining that the remaining preset log files that have not been scanned completely except the preset log file include the second archive log; The target Bloom filter is a Bloom filter corresponding to other preset log files that have not been scanned except the preset log file.

[0010] Optionally, before recording the transaction identifier of the data operation transaction corresponding to each archive log in the preset log file to the Bloom filter corresponding to the preset log file, the method further includes: Obtaining the data synchronization delay time between the first database and the second database at the current moment, and the scanning time of the archive log record corresponding to the latest scanned data operation transaction in history, and obtaining at least one historical log file corresponding to the preset log file, wherein the historical log file is a preset log file scanned within a preset historical time period; According to the data synchronization delay time, the scanning time and the at least one historical log file, parameters of the Bloom filter corresponding to the preset log file are configured to determine the capacity corresponding to the Bloom filter and the number of hash functions.

[0011] Optionally, the configuring parameters of the Bloom filter corresponding to the preset log file according to the data synchronization delay time, the scanning time and the at least one historical log file includes: Obtain the number of times each historical log file scanned within a preset historical time period has been merged, scanned and parsed and the number of transactions contained in each file; Determine the estimated number of times the preset log file is subjected to merge scanning and parsing according to the number of times each historical log file is subjected to merge scanning and parsing; Determine the estimated tolerable false alarm rate corresponding to the preset log file according to the data synchronization delay time, the scanning time, the estimated number of times and a preset maximum tolerable synchronization delay threshold; Determine a predicted value of the number of transactions corresponding to the preset log file according to the number of transactions respectively contained in each historical log file; Determining the capacity of the Bloom filter and the number of hash functions corresponding to the preset log file according to the estimated tolerable false alarm rate and the predicted value of the number of transactions; According to the capacity and the number of the hash functions, parameters of the Bloom filter corresponding to the preset log file are configured.

[0012] In order to achieve the above object, a second aspect of the present disclosure provides a data processing device, the device comprising: A first acquisition module, used to acquire a plurality of archive logs corresponding to a first database, wherein the archive logs are used to record data change operations of the first database; A file generation module, used to generate a plurality of preset log files, each of which stores at least one archived log; A second acquisition module is used to acquire, for each of the preset log files, a transaction identifier of a data operation transaction corresponding to each of the archive logs in the preset log file, wherein the data operation transaction is a logical work unit of a group of data change operations; wherein, for each of the data operation transactions, the data operation transaction corresponds to multiple archive logs, and the multiple archive logs corresponding to the data operation transaction are from the same or different preset log files; A log parsing module, used to perform multi-threaded parallel scanning and parsing on a plurality of the preset log files, and obtain the parsing result of each archived log; A data synchronization module is used to send each of the data operation transactions to a target terminal according to the transaction identifier and parsing result of each archive log. The target terminal is used to perform the data change operation on a second database according to the data operation transaction. The second database is a database that performs data synchronization processing with the first database.

[0013] In order to achieve the above-mentioned purpose, the third aspect of the present disclosure provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the first aspect of the present disclosure are implemented.

[0014] In order to achieve the above-mentioned object, a fourth aspect of the present disclosure provides an electronic device, including: a memory having a computer program stored thereon; A processor is used to execute the computer program in the memory to implement the steps of the method described in the first aspect of the present disclosure.

[0015] Through the above technical solution, multiple archive logs corresponding to the first database are obtained, and the archive logs are used to record the data change operations of the first database; multiple preset log files are generated, and at least one archive log is stored in each preset log file; for each preset log file, the transaction identifier of the data operation transaction corresponding to each archive log in the preset log file is obtained, and the data operation transaction is a logical working unit of a group of data change operations; for each data operation transaction, the data operation transaction corresponds to multiple archive logs, and the multiple archive logs corresponding to the data operation transaction are from the same or different preset log files; multi-threaded parallel scanning and parsing of the multiple preset log files is performed to obtain the transaction identifier of each archive log Parsing result: According to the transaction identifier and parsing result of each archive log, each data operation transaction is sent to the target terminal respectively, and the target terminal is used to perform the data change operation on the second database according to the data operation transaction. The second database is a database that performs data synchronization processing with the first database. In this way, by storing each archive log in a corresponding preset log file (multiple logs can be stored in one file), and then performing multi-threaded parallel scanning and parsing on multiple preset log files, the problem of data processing accumulation and delay caused by the upstream data generation speed being significantly higher than the intermediate processing speed when the single-threaded archive log scanning and parsing is performed is solved, thereby improving the data processing speed of the intermediate links and thereby improving the data synchronization efficiency.

[0016] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings: Figure 1 It is a schematic diagram of sequentially scanning archive logs provided in the related art.

[0018] Figure 2 The invention is a flow chart of a data processing method according to an exemplary embodiment.

[0019] Figure 3 is based on Figure 2 The illustrated embodiment shows a flow chart of a data processing method.

[0020] Figure 4 is based on Figure 2 The illustrated embodiment shows a flow chart of a data processing method.

[0021] Figure 5 is based on Figure 4 The illustrated embodiment shows a flow chart of a data processing method.

[0022] Figure 6 It is a block diagram of a data processing device according to an exemplary embodiment.

[0023] Figure 7 is based on Figure 6 The illustrated embodiment shows a block diagram of a data processing device.

[0024] Figure 8 is based on Figure 6 The illustrated embodiment shows a block diagram of a data processing device.

[0025] Fig. 9 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0026] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0027] First, the application scenario of the present disclosure is explained. The present disclosure is mainly used in the scenario of data synchronization or data replication of the database. For example, a data change operation (including insertion, deletion, modification, etc.) is performed on database A (for example, an Oracle database), and the data change operation in database A needs to be synchronized to database B. Among them, the change operation on the database involves the concept of a "transaction". The logical working unit of a group of data change operations (such as insertion, update, deletion, etc.) is called a transaction. The group of data change operations corresponding to the same transaction must all be successfully executed, or all fail and roll back, so as to ensure the consistency and integrity of the data change operation. The archive log generated by an Oracle transaction is usually composed of the following parts according to the log type classification.

[0028] (1) Transaction Start Record: Indicates the start of a transaction and records relevant information of the transaction, such as transaction ID, start time, etc.

[0029] (2) Data modification log (Redo Record): records a set of specific data change operations corresponding to the transaction, including operations such as adding, updating or deleting data rows, as well as the corresponding data row information.

[0030] (3) Transaction Commit Record: Indicates that the transaction has been committed and records relevant information about the transaction commit, such as transaction ID, commit time, etc.

[0031] In order to achieve data synchronization, a single-threaded file scanner is provided in the related art. The single-threaded file scanner can scan the archive log file list in the order of the SCN (System Change Number) of the archive log corresponding to database A, and load the archive log line by line in sequence. Among them, SCN can be understood as the identification code of each archive log, and different archive logs are identified by different SCNs. For example, SCN1, SCN2, SCN3, etc., respectively represent different archive logs. Figure 1 It is a schematic diagram of sequentially scanning archive logs provided in the related art, such as Figure 1 As shown, when a transaction start log is encountered, a cache space for the transaction ID is created in the event manager to be sent; when a data modification log is encountered, the cache space corresponding to the transaction ID is found, and the data change operation recorded in the data modification log is inserted into the cache queue corresponding to the transaction ID; when a transaction commit log is encountered, it is marked that the archive log corresponding to this transaction ID has been completely scanned, and the data operation transaction corresponding to the transaction ID is sent downstream to the target terminal (such as the terminal corresponding to database B).

[0032] However, the upstream generates archive logs through multiple threads, and the intermediate link scans the archive log files through a single thread. When the upstream data generation speed is significantly higher than the intermediate processing speed, data processing will accumulate and be delayed, resulting in low overall execution efficiency, and it is impossible to improve the data processing speed of the intermediate link through performance tuning and other measures.

[0033] In order to solve the above problems, the present invention provides a data processing method, device, storage medium and electronic device. The specific implementation of the present invention is described in detail below with reference to the accompanying drawings.

[0034] Figure 2 is a flow chart of a data processing method according to an exemplary embodiment. Figure 2 As shown, the method comprises the following steps: In step S101, a plurality of archive logs corresponding to a first database are obtained, where the archive logs are used to record data change operations of the first database.

[0035] The first database here refers to a database for performing data change operations. The type of the first database may be, for example, an Oracle database, and the data change operations may include, for example, data insertion, deletion, and modification operations. In actual application scenarios, the data change operations in the first database need to be synchronized to the second database so that the data in the first database and the second database are consistent.

[0036] The archive log is a log file that records every data change operation that occurs in the database when the database is running, including insert, update, delete, etc. In the process of database replication and synchronization, the archive log can help ensure the consistency of data between different database instances. In addition, different archive logs can be identified by SCN codes, and archive logs with different SCN codes can correspond to multiple different types of archive logs of the same transaction, or archive logs of different transactions. Among them, the log types of the archive log can include transaction start logs (for example, the type identifier can be represented by START), data modification logs (such as logs with type identifiers of INSERT, DELETE, and UPDATE, respectively), and transaction commit logs (the type identifier can be COMMIT, for example). Among them, the transaction start log indicates the start of a transaction and records the relevant information of the transaction, such as transaction ID, start time, etc.; the data modification log records a set of specific data change operations performed by the current transaction, including operations such as adding, updating or deleting data rows, and corresponding data row information; the transaction commit log indicates that the current transaction has been committed, and records the relevant information of the current transaction commit, such as transaction ID, commit time, etc.

[0037] In step S102, a plurality of preset log files are generated, each of which stores at least one archived log.

[0038] In the present disclosure, multiple preset log files can be generated, and a specified number of archive logs can be saved in each preset log file, so that a preset number of preset log files can be selected for parallel scanning and parsing. For example, the specified number can be 20, so that the archive logs corresponding to SCN0-SCN19 can be stored in preset log file 1, the archive logs corresponding to SCN20-SCN39 can be stored in preset log file 2, and the archive logs corresponding to SCN40-SCN59 can be stored in preset log file 3, and so on, and the archive logs corresponding to each preset log file can be determined.

[0039] Therefore, in this step, for each archive log, the target log file corresponding to the archive log can be determined from the plurality of preset log files based on the SCN code of the archive log, and then the archive log is stored in the target log file.

[0040] In step S103, for each of the preset log files, a transaction identifier of a data operation transaction corresponding to each of the archive logs in the preset log file is obtained, where the data operation transaction is a logical work unit of a group of data change operations.

[0041] The data operation transaction here is the "transaction" mentioned above. As mentioned above, the archive log is used to record the data change operation on the first database. The logical work unit of a set of data change operations (such as insert, update, delete, etc.) is called a transaction. The set of data change operations corresponding to the same transaction must all be successfully executed, or all fail and roll back, so as to ensure the consistency and integrity of the data change operation. The archive log generated by an Oracle transaction usually includes a transaction start log, a data modification log, and a transaction commit log. In other words, a data operation transaction corresponds to multiple archive logs.

[0042] For example, a data operation transaction with a transaction ID of 0001.00A.00000321 may include the following four archive logs: 0,......,0001.00A.00000321,......,START,...... 2,......,0001.00A.00000321,......,INSERT,...... 3,......,0001.00A.00000321,......,DELETE,...... 4,......,0001.00A.00000321,......,COMMIT,...... As shown in this example, each archive log contains the SCN code of the archive log (0, 2, 3, 4 respectively), the transaction identifier 0001.00A.00000321 corresponding to the archive log, and the type identifier of the archive log (such as START, INSERT, DELETE, and COMMIT), where the type identifier is used to characterize the log type of the corresponding archive log. In this way, by parsing each archive log, the transaction identifier and log type corresponding to the archive log can be obtained.

[0043] It should be noted that each data operation transaction in the present disclosure corresponds to multiple archive logs, and the multiple archive logs corresponding to a data operation transaction come from the same or different preset log files. If the multiple archive logs corresponding to a data operation transaction come from different multiple preset log files, the archive logs corresponding to the same data operation transaction in multiple preset log files can be fragmented and scanned to ensure that all data change operations of the same data operation transaction are sent to the target terminal together, thereby ensuring the consistency of data synchronization between different databases.

[0044] In step S104, a plurality of the preset log files are scanned and parsed in parallel using multiple threads to obtain a parsing result of each archive log.

[0045] In one possible implementation, the amount of parallelism (i.e., a preset number) for multi-threaded parallel scanning of the multiple preset log files can be determined based on the hardware configuration of the execution device (such as a terminal or a server) of the data processing method. For example, the determined amount of parallelism can be expressed as N. In this way, N log parsers can be started at one time, each log parser corresponds to a preset log file, and each log parser can scan and parse the archived logs in the corresponding preset log file line by line, thereby realizing the multi-threaded parallel scanning and parsing of N preset log files using N log parsers.

[0046] In addition, the parsing result may include the log type of each archive log, and the log type may include a transaction start log, a data modification log, or a transaction commit log.

[0047] That is to say, in this step, a preset number of log parsers can be started; the preset number of log parsers correspond one-to-one to the preset number of preset log files; for each preset log file in the preset number of preset log files, each archived log in the preset log file is scanned line by line by the target log parser corresponding to the preset log file; for the first archived log currently scanned, after parsing the first archived log by the target log parser, the log type of the first archived log can be obtained.

[0048] In step S105, each of the data operation transactions is sent to a target terminal according to the transaction identifier and parsing result of each archive log. The target terminal is used to perform the data change operation on a second database according to the data operation transaction. The second database is a database that performs data synchronization processing with the first database.

[0049] Wherein, the second database is a relational database.

[0050] By adopting the above method, each archived log is stored in the corresponding preset log file (multiple logs can be stored in one file), and then the multiple preset log files are scanned and analyzed in parallel by multiple threads. This solves the problem of data processing accumulation and delay caused by the upstream data generation speed being significantly higher than the intermediate processing speed when the archived log is scanned and analyzed by a single thread, thereby improving the data processing speed in the intermediate links and thereby improving the data synchronization efficiency.

[0051] Figure 3 is based on Figure 2 The embodiment shown is a flow chart of a data processing method, such as Figure 3 As shown, step S104 includes the following sub-steps: In step S1041, a preset number of log parsers are started, and the preset number of log parsers corresponds one-to-one to the preset number of preset log files.

[0052] In step S1042, for each preset log file in the preset number of preset log files, each archive log in the preset log file is scanned line by line by a target log parser corresponding to the preset log file.

[0053] In step S1043, for the first archive log currently scanned, the first archive log is parsed by the target log parser to obtain the log type of the first archive log.

[0054] For example, a log parser scans and parses the preset log file corresponding to it as an example, and the log parser scans multiple archive logs in the preset log file line by line, and performs the same parsing steps for each archive log scanned. Taking the first archive log currently scanned as an example, after parsing the first archive log, the type identification corresponding to the first archive log is determined, and the log type of the first archive log can be determined based on the type identification. For example, if the type identification is START, it means that the log type of the first archive log is a transaction start log, if the type identification is one of INSERT, DELETE, and UPDATE, it means that the log type of the first archive log is a data modification log, and if the type identification is COMMIT, it means that the log type of the first archive log is a transaction commit log. The above examples are only illustrative, and the present disclosure is not limited to this.

[0055] In addition, as described above, by parsing each archive log, the transaction identifier of the data operation transaction corresponding to each archive log can also be obtained. In this way, after obtaining the transaction identifier and log type corresponding to the first archive log, the data operation transaction corresponding to the transaction identifier can be sent to the target terminal according to the transaction identifier and the log type.

[0056] Figure 4 is based on Figure 2 The embodiment shown is a flow chart of a data processing method, such as Figure 4 As shown, step S105 includes the following sub-steps: In step S1051: for the first archive log currently scanned in each archive log, according to the transaction identifier and log type of the first archive log, the data change operation corresponding to the first archive log is stored in the target cache space corresponding to the target transaction identifier; wherein the target transaction identifier is the transaction identifier of the data operation transaction corresponding to the first archive log.

[0057] In addition to transaction commit logs, this log type may also include transaction start logs or data modification logs, where the transaction start log represents the start of the corresponding data operation transaction, the data modification log represents the data change operation of the corresponding data operation transaction, and the transaction commit log is used to indicate the end of the corresponding data operation transaction.

[0058] In a possible implementation of this step, it can be determined first whether the target cache space corresponding to the target transaction identifier has been created in the preset cache space; if the target cache space has not been created in the preset cache space, if the log type of the first archive log is a transaction start log, the target cache space can be created in the preset cache space; if the log type of the first archive log is a data modification log, after the target cache space is created in the preset cache space, the data change operation recorded in the first archive log can be stored in the target cache space; if the log type of the first archive log is a transaction commit log, the target cache space can be created in the preset cache space; that is, in the case of determining that the target cache space corresponding to the target transaction identifier has not been created in the preset cache space, for the first archive log of any log type currently scanned, the target cache space corresponding to the target transaction identifier can not be created in the preset cache space. Among them, the target transaction identifier of the first archive log can be used to determine whether the target cache space corresponding to the target transaction identifier has been created in the preset cache space.

[0059] In addition, when the target cache space has been created in the preset cache space, if the log type of the first archive log is a data modification log, the data change operations recorded in the first archive log can be directly stored in the target cache space; if the log type of the first archive log is a transaction commit log, it can be determined whether all data change operations under the data operation transaction corresponding to the target transaction identifier (that is, the transaction identifier corresponding to the first archive log) have been stored in the target cache space, and when it is determined that all data change operations under the data operation transaction corresponding to the target transaction identifier are stored in the target cache space, the data operation transaction corresponding to the target cache space can be sent to the target terminal. In the process of judging whether the target cache space has stored all the data change operations under the data operation transaction corresponding to the target transaction identifier, it can be judged whether the remaining preset log files that have not been scanned, except the preset log file currently being scanned, contain the second archive log corresponding to the target transaction identifier. When it is determined that the remaining preset log files that have not been scanned do not contain the second archive log, it can be determined that the target cache space has stored all the data change operations under the data operation transaction corresponding to the target transaction identifier; when it is determined that the remaining preset log files that have not been scanned contain the second archive log, it is necessary to store the data change operations corresponding to the second archive log in the target cache space corresponding to the target transaction identifier (that is, the "cross-file fragment merging scanning and parsing" mentioned later), and then it is considered that the target cache space has stored all the data change operations under the data operation transaction corresponding to the target transaction identifier. The second archive log is the archive log corresponding to the target transaction identifier that has not been scanned except the first archive log.

[0060] The preset buffer space may include a preset buffer space of a transaction manager to be sent, and the transaction manager to be sent is used to manage data operation transactions.

[0061] For example, assuming that the target transaction identifier corresponding to the first archive log currently scanned is 0001.00A.00000321, it is first determined whether the target cache space corresponding to the target transaction identifier 0001.00A.00000321 has been created in the preset cache space of the transaction manager to be sent. When it is determined that the target cache space has not been created in the preset cache space, if the first archive log is a transaction start log, the target cache space corresponding to the target transaction identifier 0001.00A.00000321 can be created in the preset cache space of the transaction manager to be sent. If the first archive log is a data modification log, after the target cache space corresponding to the target transaction identifier 0001.00A.00000321 is created in the preset cache space, the data change operation recorded in the first archive log is stored in the target cache space. If the first archive log is a transaction commit log, it is also necessary to create the target cache space corresponding to the target transaction identifier 0001.00A.00000321 in the preset cache space of the transaction manager to be sent. In the case where it is determined that the target cache space corresponding to the target transaction identifier 0001.00A.00000321 has been created in the preset cache space, if the log type of the first archive log is a data modification log, the data change operation recorded in the first archive log can be directly stored in the target cache space; if the log type of the first archive log is a transaction commit log, it can be determined first whether other unfinished scanned preset log files except the preset log file where the first archive log is located include other unfinished scanned files except the first archive log corresponding to the target transaction identifier 0001.00A.00000321 When it is determined that the second archive log corresponding to the target transaction identifier 0001.00A.00000321 is included in other preset log files that have not been scanned, the data change operation corresponding to the second archive log can be stored in the target cache space corresponding to the target transaction identifier 0001.00A.00000321, and then the data operation transaction with the target transaction identifier 0001.00A.00000321 corresponding to the target cache space can be sent to the target terminal. When it is determined that the second archive log corresponding to the target transaction identifier 0001.00A.00000321 is not included in other preset log files that have not been scanned, it indicates that all data change operations under the data operation transaction with the target transaction identifier 0001.00A.00000321 have been stored in the target cache space, and the data operation transaction with the target transaction identifier 0001.00A.00000321 can be sent to the target terminal. The examples here are merely illustrative and the present disclosure is not limited thereto.

[0062] It should be noted that when enabling multi-threading to perform parallel scanning of archive logs, for multiple archive logs of the same transaction, when the events of the archive logs generated later that are scanned first are sent downstream to the target terminal before the events of the archive logs generated earlier, it will cause data inconsistency between the target terminal and the source terminal.

[0063] For example, suppose that a multi-threaded parallel scan of archive logs is started. If there are two parallel tasks and the data volume of a single task is 20, the 20 archive logs corresponding to SCN codes 0-19 (corresponding to task 1) and the 20 archive logs corresponding to SCN codes 20-39 (corresponding to task 2) are respectively executed in parallel as two scanning tasks, and each task loads the archive logs it is responsible for line by line. The multiple archive logs contained in a transaction are: 0,......,0001.00A.00000321,......,START,...... 2,......,0001.00A.00000321,......,INSERT,...... 3,......,0001.00A.00000321,......,INSERT,...... 6,......,0001.00A.00000321,......,UPDATE,...... 20,......,0001.00A.00000321,......,INSERT,...... 21,......,0001.00A.00000321,......,COMMIT,...... Based on the above two parallel tasks, the archive logs with SCN codes of 0, 2, 3, and 6 in the archive log corresponding to the transaction (transaction ID is 0001.00A.00000321) are assigned to task 1 for scanning, and the archive logs with SCN codes of 20 and 21 are assigned to task 2 for scanning. Since task 1 and task 2 are executed in parallel, the archive log with SCN code 21 in task 2 (the log is the transaction commit log) may be scanned before the archive logs with SCN codes of 3 and 6 in task 1. In the related art, after scanning the transaction commit log, the data change operation corresponding to the transaction is sent to the target terminal. At this time, the archive logs of the same transaction in task 1 only scan the two logs with SCN codes of 0 and 2, and the logs with SCN codes of 3 and 6 have not been scanned. Since the corresponding COMMIT record cannot be scanned later, the data change operations of the two archive logs with SCN codes of 3 and 6 will be discarded and cannot be sent to the target terminal, resulting in inconsistent data between the target terminal and the source terminal.

[0064] Therefore, in order to solve the problem of inconsistent data between the target terminal and the source end during the multi-threaded parallel scanning of multiple archive logs of the first database, the present disclosure can perform cross-file fragment merging scanning and parsing on multiple preset log files where multiple archive logs under the same transaction identifier are located. Specifically, the log parser scans and parses the archive logs in the preset log file corresponding to it line by line, and when it is determined that the log type of the scanned first archive log is a transaction commit log, it can be determined whether other archive logs of the target transaction identifier corresponding to the transaction commit log (i.e., other archive logs that have not been scanned in full except the first archive log, which can be called "second archive logs") appear in the remaining preset log files that have not been scanned in full. If so, in order to avoid the problem of inconsistent data between the target terminal and the source end, the multiple preset log files hit by the target transaction identifier can be subjected to fragment merging scanning and processing of the archive logs under the same transaction, so as to store all data change operations under the data operation transaction corresponding to the target transaction identifier in the target cache space, and then send all data change operations under the data operation transaction stored in the target cache space to the target terminal together, thereby avoiding the problem of inconsistent data between the target terminal and the source end.

[0065] In step S1052, when all data change operations under the data operation transaction corresponding to the target transaction identifier are stored in the target cache space, the data operation transaction corresponding to the target cache space is sent to the target terminal.

[0066] The log type may also include a transaction commit log, and the transaction commit log is used to indicate the end of the corresponding data operation transaction.

[0067] In a possible implementation of this step, when the log type of the first archive log is a transaction commit log, it is determined whether the remaining preset log files of the unfinished scan except the preset log file contain the second archive log corresponding to the target transaction identifier; when it is determined that the remaining preset log files of the unfinished scan except the preset log file contain the second archive log, the data change operation corresponding to the second archive log can be stored in the target cache space corresponding to the target transaction identifier, and then the data operation transaction corresponding to the first archive log is sent to the target terminal; when it is determined that the remaining preset log files of the unfinished scan except the preset log file do not contain the second archive log, the data operation transaction corresponding to the first archive log is sent to the target terminal; wherein, the second archive log is the archive log of the unfinished scan except the first archive log corresponding to the target transaction identifier.

[0068] It can be understood that, in the case where it is determined that the second archive log is included in the preset log files that have not been scanned in full except the preset log files, the data change operation corresponding to the second archive log is stored in the target cache space corresponding to the target transaction identifier, so that all data change operations under the data operation transaction corresponding to the first archive log can be stored in the target cache space, so that all data change operations under the data operation transaction corresponding to the first archive log stored in the target cache space can be sent to the target terminal together. In the case where it is determined that the second archive log is not included in the preset log files that have not been scanned in full except the preset log files, it means that the archive log corresponding to the target transaction identifier only appears in the current preset log file, and the archive log corresponding to the target transaction identifier has been completely scanned, and the data operation transaction corresponding to the first archive log can be sent to the target terminal. In this way, the problem of data inconsistency between the first database and the second database caused by sending only part of the data change operations under the data operation transaction to the target terminal can be avoided.

[0069] In addition, in the process of executing this step, when the log type of the first archive log is a transaction commit log, it can be determined in the following way whether the remaining preset log files that have not been scanned completely except the preset log file contain the second archive log corresponding to the target transaction identifier: a Bloom filter corresponding to each preset log file is configured (that is, each preset log file is configured with a Bloom filter corresponding to it), and the Bloom filter is used to characterize the set of transaction identifiers of the archive logs in the corresponding preset log file. In other words, the Bloom filter is used to record the transaction identifiers of the archive logs stored in the preset log file corresponding to the Bloom filter. For each preset log file, the transaction identifiers of the data operation transactions corresponding to each archive log in the preset log file can be recorded to the Bloom filter corresponding to the preset log file. In this way, in the process of executing step S1052, when it is determined that the target transaction identifier is located in the target Bloom filter, it can be determined that the remaining preset log files that have not been scanned completely except the preset log file contain the second archive log; wherein, the target Bloom filter is the Bloom filter corresponding to the remaining preset log files that have not been scanned completely except the preset log file.

[0070] For example, it is assumed that two log parsers are started to perform parallel parsing processing on two preset log files, wherein the preset log file 1 stores the following archive logs: 0,......,0001.00A.00000321,......,START,...... 2,......,0001.00A.00000321,......,INSERT,...... 3,......,0001.00A.00000321,......,INSERT,...... 6,......,0001.00A.00000321,......,UPDATE,...... The following archive logs are stored in the preset log file 2: 20,......,0001.00A.00000321,......,INSERT,...... 21,......,0001.00A.00000321,......,COMMIT,...... 22,......,0001.00A.00000322,......,START,...... 23,......,0001.00A.00000322,......,INSERT,...... That is to say, there are currently two log scanning and parsing threads (thread 1 and thread 2) executing in parallel, where the log parser of thread 1 scans and parses the archive logs in the preset log file 1 line by line, and the log parser of thread 2 scans and parses the archive logs in the preset log file 2 line by line. And each preset log file has a corresponding Bloom filter that records the transaction identifiers in the preset log file. For preset log file 2, the Bloom filter corresponding to preset log file 1 is the target Bloom filter. For thread 2, when scanning the archive log with SCN code 21, it can be determined through analysis that the archive log is a transaction commit log, and the transaction identifier of the transaction commit log is 0001.00A.00000321. It is judged whether the transaction identifier 0001.00A.00000321 is located in the target Bloom filter corresponding to the preset log file 1. Since the preset log file 1 also stores the archive log corresponding to the transaction identifier 0001.00A.00000321, the transaction identifier 0001.00A.00000321 is also located in the target Bloom filter corresponding to the preset log file 1. After that, it can be determined from the preset log file 1 corresponding to the target Bloom filter that the unscanned second archive log corresponding to the transaction identifier 0001.00A.00000321 includes the archive logs with SCN codes 3 and 6 (which can be understood as the “logs” that have not been scanned). Then, the data change operations of the archive log records with SCN codes 3 and 6 in the preset log file 1 (i.e., the insert operation corresponding to INSERT and the update operation corresponding to UPDATE) can be stored in the target cache space corresponding to the transaction identifier 0001.00A.00000321. In this way, the target cache space corresponding to the transaction identifier 0001.00A.00000321 stores all the data change operations of the data operation transaction corresponding to the transaction identifier 0001.00A.00000321 (including four data change operations of the archive log records with SCN codes 2, 3, 6, and 20 respectively). Then, all the data change operations corresponding to the transaction identifier 0001.00A.00000321 can be sent to the target terminal, thereby avoiding the problem that the data change operations of the two archive logs with SCN codes 3 and 6 are discarded and cannot be sent to the target terminal due to the logs with SCN codes 3 and 6 in the preset log file 1 not being scanned, thereby causing the problem of data inconsistency between the target terminal and the source terminal. The above examples are merely illustrative and the present disclosure is not limited thereto.

[0071] By adopting the above method, during the multi-threaded parallel scanning of multiple archive logs corresponding to the first database, the integrity and sequential consistency of the data change operation under a data operation transaction sent downstream are guaranteed by merging and scanning the fragments across files.

[0072] The following describes the parameter configuration process of the Bloom filter configured for each preset log file. Bloom filter is an efficient probabilistic data structure used to determine whether an element belongs to a set. Bloom filter determines whether an element is in a set by using capacity and multiple hash functions. Among them, capacity refers to the size of the bit array, that is, the number of elements that can be accommodated. Generally, the larger the capacity of the Bloom filter, the more elements can be stored. The selection of capacity should be determined based on the number of elements that need to be stored in the actual application scenario and the allowed false positive rate. The number of hash functions refers to the number of hash functions used in the Bloom filter. The role of the hash function is to map elements to different positions in the bit array and is used to set the bit value in the bit array. Generally, the more hash functions there are, the better the performance of the Bloom filter and the lower the false positive rate. The number of hash functions should be selected based on the number of elements in the actual application and the expected false positive rate, and appropriate trade-offs should be made.

[0073] It should be noted that there is a certain correlation between the capacity of the Bloom filter and the number of hash functions. Increasing the capacity may require increasing the number of hash functions to maintain a certain false positive rate. At the same time, too many hash functions will also increase the computational overhead. Therefore, when designing a Bloom filter, it is necessary to comprehensively consider the capacity and the number of hash functions to meet actual needs.

[0074] In the present disclosure, the capacity of the Bloom filter and the number of hash functions corresponding to each preset log file except the first preset log file can be configured in the following manner, wherein the capacity of the Bloom filter and the number of hash functions corresponding to the first preset log file can be empirical values, and the present disclosure does not make any specific limitations on this.

[0075] Figure 5 is based on Figure 4 The embodiment shown is a flow chart of a data processing method, such as Figure 5 As shown, the method further comprises the following steps: In step S106, the data synchronization delay time between the first database and the second database at the current moment and the scanning time of the archived log record corresponding to the latest scanned data operation transaction in history are obtained, and at least one historical log file corresponding to the preset log file is obtained, and the historical log file is a preset log file scanned within a preset historical time period.

[0076] In step S107, parameters of the Bloom filter corresponding to the preset log file are configured according to the data synchronization delay time, the scanning time and the at least one historical log file to determine the capacity corresponding to the Bloom filter and the number of hash functions.

[0077] Among them, in the process of configuring the parameters of the Bloom filter corresponding to the preset log file according to the data synchronization delay time, the scanning time and the at least one historical log file, the number of times each historical log file scanned within the preset historical time period is merged, scanned and analyzed and the number of transactions contained in each file can be obtained; according to the number of times each historical log file is merged, scanned and analyzed, the estimated number of times the preset log file is merged, scanned and analyzed is determined; according to the data synchronization delay time, the scanning time, the estimated number of times and the preset maximum tolerable synchronization delay threshold, the estimated tolerable false alarm rate corresponding to the preset log file is determined; according to the number of transactions contained in each historical log file, the predicted value of the number of transactions corresponding to the preset log file is determined; according to the estimated tolerable false alarm rate and the predicted value of the number of transactions, the capacity of the Bloom filter corresponding to the preset log file and the number of hash functions are determined; according to the capacity and the number of hash functions, the parameters of the Bloom filter corresponding to the preset log file are configured.

[0078] Among them, in the process of determining the capacity of the Bloom filter corresponding to the preset log file and the number of hash functions according to the estimated tolerable false alarm rate and the predicted value of the number of transactions, the capacity of the Bloom filter corresponding to the preset log file can be determined according to the estimated tolerable false alarm rate and the predicted value of the number of transactions; the number of hash functions of the Bloom filter corresponding to the preset log file can be determined according to the predicted value of the number of transactions and the capacity.

[0079] It should be noted that the data processing method in the present disclosure performs real-time scanning and parsing as the archive logs corresponding to the first database are generated. Before the archive logs are stored in the corresponding preset log files, the parameters of the Bloom filter of the preset log files need to be configured first, and the parameters of the Bloom filter of the current latest preset log file can be configured based on the historical scanning results of the historical log files that have been scanned within the preset historical time period.

[0080] For example, taking the configuration of parameters of the Bloom filter corresponding to the latest preset log file (the preset log file that has not yet stored the archived log) as an example, the configuration process includes the following steps: a. Obtain at least one historical log file corresponding to the preset log file (the historical log file refers to the preset log file scanned within the preset historical time period, and the end time of the preset historical time period is the current time), and then for each of the historical log files, obtain the number of times the historical log file is merged, scanned and parsed and the number of transactions contained in each. Then calculate the average number of times the at least one historical log file is merged, scanned and parsed within the preset historical time period, and use the average number as the estimated number of times the preset log file to be configured at the current time is merged, scanned and parsed, and the calculation formula is: (1) Among them, aveMSC represents the average number of times (that is, the estimated number of times the preset log file to be configured at the current moment is merged, scanned and parsed), n represents the number of historical log files scanned within the preset historical time period, and MSCi (i=1,2,...,n) represents the number of times the i-th historical log file is merged, scanned and parsed within the preset historical time period.

[0081] b. Obtain the data synchronization delay time between the first database and the second database at the current moment, and the scanning time of the archive log record corresponding to the latest scanned data operation transaction in history. In this way, the estimated tolerance false alarm rate corresponding to the preset log file can be determined based on the data synchronization delay time, the scanning time, the estimated number of times, and the preset maximum tolerance synchronization delay threshold. The calculation formula is: (2) in, Indicates the false alarm tolerance time, Indicates the estimated number of false alarms. Indicates the data synchronization delay time. Indicates the maximum tolerated synchronization delay threshold, Indicates the scan time of the archive log record corresponding to the latest scanned data operation transaction in history. represents the estimated tolerable false alarm rate, Indicates the system default maximum false alarm rate.

[0082] c. Determine the predicted value of the number of transactions corresponding to the preset log file according to the number of transactions contained in each historical log file. The specific calculation formula is: (3) in, Indicates the predicted value of the number of transactions corresponding to the i+1th preset log file (i.e., the preset log file currently to be configured), Indicates the actual number of transactions in the i-th preset log file (i.e., the log file in the historical log file whose scan time is closest to the current time). represents the predicted value of the number of transactions corresponding to the i-th preset log file, represents the smoothing factor, and The value range is ,For example, =0.98, n represents the number of historical log files scanned within the preset historical time period.

[0083] It should be noted that the transaction numbers contained in the preset log files adjacent in time are similar, so the transaction number contained in the historical log file whose scanning time is closest to the current moment has the greatest impact on the estimated transaction number of the current preset log file.

[0084] d. According to the estimated tolerable false alarm rate and the predicted value of the number of transactions, the capacity of the Bloom filter corresponding to the preset log file is determined, and the calculation formula is: (4) Where m represents the capacity of the Bloom filter corresponding to the preset log file to be configured. represents the predicted value of the number of transactions corresponding to the preset log file calculated based on the above formula (3), It represents the estimated tolerable false alarm rate calculated based on formula (2).

[0085] e. According to the predicted value of the number of transactions and the capacity, determine the number of hash functions of the Bloom filter corresponding to the preset log file, and the calculation formula is: (5) in, Represents the number of hash functions, represents the predicted value of the number of transactions corresponding to the preset log file calculated based on the above formula (3), and m represents the capacity of the Bloom filter corresponding to the preset log file to be configured calculated based on formula (4).

[0086] The above examples are merely illustrative and the present disclosure is not limited thereto.

[0087] By adopting the above method, the parameter configuration of the Bloom filter corresponding to each preset log file can be realized. In this way, during the merge scanning process of log fragments, the Bloom filter is used as a mechanism for searching and filtering according to the transaction identifier, which improves the overall processing efficiency; and according to the data of the historical log files that have been scanned (such as delay, merge scanning times and other indicators), the capacity of the Bloom filter and the number of hash functions are dynamically adjusted, so that the overall performance of the Bloom filter can be adjusted to the optimal configuration, reducing the impact on the entire data synchronization process.

[0088] Figure 6 is a block diagram of a data processing device according to an exemplary embodiment. Figure 6 As shown, the device comprises: A first acquisition module 601 is used to acquire a plurality of archive logs corresponding to a first database, where the archive logs are used to record data change operations of the first database; A file generation module 602, used to generate a plurality of preset log files, each of which stores at least one archived log; The second acquisition module 603 is used to acquire, for each of the preset log files, a transaction identifier of a data operation transaction corresponding to each of the archive logs in the preset log file, where the data operation transaction is a logical work unit of a group of data change operations; wherein, for each of the data operation transactions, the data operation transaction corresponds to multiple archive logs, and the multiple archive logs corresponding to the data operation transaction are from the same or different preset log files; The log parsing module 604 is used to perform multi-threaded parallel scanning and parsing on the plurality of preset log files to obtain the parsing result of each archived log; The data synchronization module 605 is used to send each of the data operation transactions to the target terminal according to the transaction identifier and parsing result of each archive log. The target terminal is used to perform the data change operation on the second database according to the data operation transaction. The second database is a database that performs data synchronization processing with the first database.

[0089] Optionally, the parsing result includes the log type of each archived log; the log parsing module 604 is used to start a preset number of log parsers, and the preset number of log parsers corresponds to the preset number of preset log files one by one; for each preset log file in the preset number of preset log files, each archived log in the preset log file is scanned line by line by a target log parser corresponding to the preset log file; for the first archived log currently scanned, the log type of the first archived log is obtained after the first archived log is parsed by the target log parser; The data synchronization module 605 is used to store the data change operation corresponding to the first archive log currently scanned in each of the archive logs into the target cache space corresponding to the target transaction identifier according to the transaction identifier and log type of the first archive log; wherein the target transaction identifier is the transaction identifier of the data operation transaction corresponding to the first archive log; and when all data change operations under the data operation transaction corresponding to the target transaction identifier are stored in the target cache space, the data operation transaction corresponding to the target cache space is sent to the target terminal.

[0090] Optionally, the log type includes a transaction commit log, which is used to indicate the end of a corresponding data operation transaction; the data synchronization module 605 is used to determine whether the remaining preset log files of unfinished scanning other than the preset log file contain the second archive log corresponding to the target transaction identifier when the log type of the first archive log is the transaction commit log; when it is determined that the remaining preset log files of unfinished scanning other than the preset log file contain the second archive log, after storing the data change operation corresponding to the second archive log in the target cache space corresponding to the target transaction identifier, the data operation transaction corresponding to the first archive log is sent to the target terminal; when it is determined that the remaining preset log files of unfinished scanning other than the preset log file do not contain the second archive log, the data operation transaction corresponding to the first archive log is sent to the target terminal; wherein, the second archive log is the archive log of other unfinished scanning other than the first archive log corresponding to the target transaction identifier.

[0091] Optionally, the log type further includes a transaction start log or a data modification log, wherein the transaction start log indicates the start of a corresponding data operation transaction, and the data modification log indicates the data change operation of the corresponding data operation transaction; The data synchronization module 605 is used to determine whether the target cache space corresponding to the target transaction identifier has been created in the preset cache space; if the target cache space has not been created in the preset cache space, if the log type of the first archive log is the transaction start log, the target cache space is created in the preset cache space; if the log type of the first archive log is the data modification log, after the target cache space is created in the preset cache space, the data change operation recorded in the first archive log is stored in the target cache space; if the log type of the first archive log is the transaction commit log, the target cache space is created in the preset cache space. cache space; in the case where the target cache space has been created in the preset cache space, if the log type of the first archive log is the data modification log, the data change operation recorded in the first archive log is stored in the target cache space; if the log type of the first archive log is the transaction commit log, it is determined whether all data change operations under the data operation transaction corresponding to the target transaction identifier are stored in the target cache space, and when it is determined that all data change operations under the data operation transaction corresponding to the target transaction identifier are stored in the target cache space, the data operation transaction corresponding to the target cache space is sent to the target terminal.

[0092] Optionally, each of the preset log files is configured with a corresponding Bloom filter, and the Bloom filter is used to characterize a set of transaction identifiers of archived logs in the corresponding preset log file; Figure 7 is based on Figure 6 The block diagram of a data processing device shown in the embodiment shown in FIG. Figure 7 As shown, the device also includes: A transaction identification recording module 606 is used to record, for each of the preset log files, the transaction identification of the data operation transaction corresponding to each of the archive logs in the preset log file to the Bloom filter corresponding to the preset log file; The data synchronization module 605 is used to determine that the remaining unscanned preset log files other than the preset log file contain the second archive log when it is determined that the target transaction identifier is located in the target Bloom filter; wherein the target Bloom filter is the Bloom filter corresponding to the remaining unscanned preset log files other than the preset log file.

[0093] Optionally, Figure 8 is based on Figure 6 The block diagram of a data processing device shown in the embodiment shown in FIG. Figure 8 As shown, the device also includes: The parameter configuration module 607 obtains the data synchronization delay time between the first database and the second database at the current moment, and the scanning time of the archived log record corresponding to the latest scanned data operation transaction in history, and obtains at least one historical log file corresponding to the preset log file, and the historical log file is a preset log file scanned within a preset historical time period; according to the data synchronization delay time, the scanning time and the at least one historical log file, the Bloom filter corresponding to the preset log file is parameter configured to determine the capacity corresponding to the Bloom filter and the number of hash functions.

[0094] Optionally, the parameter configuration module 607 is used to obtain the number of times each historical log file scanned within a preset historical time period is merged, scanned and parsed and the number of transactions contained in each file; determine the estimated number of times the preset log file is merged, scanned and parsed based on the number of times each historical log file is merged, scanned and parsed; determine the estimated tolerable false alarm rate corresponding to the preset log file based on the data synchronization delay time, the scanning time, the estimated number of times and a preset maximum tolerable synchronization delay threshold; determine the predicted number of transactions corresponding to the preset log file based on the number of transactions contained in each historical log file; determine the capacity of the Bloom filter corresponding to the preset log file and the number of hash functions based on the estimated tolerable false alarm rate and the predicted number of transactions; and perform parameter configuration on the Bloom filter corresponding to the preset log file based on the capacity and the number of hash functions.

[0095] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0096] Fig. 9 900 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 900 may be provided as a server. Fig. 9 The electronic device 900 includes a processor 922, which may be one or more, and a memory 932 for storing a computer program executable by the processor 922. The computer program stored in the memory 932 may include one or more modules each corresponding to a set of instructions. In addition, the processor 922 may be configured to execute the computer program to perform the above-mentioned data processing method.

[0097] In addition, the electronic device 900 may further include a power supply component 926 and a communication component 950, wherein the power supply component 926 may be configured to perform power management of the electronic device 900, and the communication component 950 may be configured to implement communication of the electronic device 900, for example, wired or wireless communication. In addition, the electronic device 900 may further include an input / output (I / O) interface 958. The electronic device 900 may operate based on an operating system stored in the memory 932, such as Windows ServerTM, Mac OSXTM, UnixTM, LinuxTM, etc.

[0098] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by the processor, the steps of the above-mentioned data processing method are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 932 including program instructions, and the above-mentioned program instructions can be executed by the processor 922 of the electronic device 900 to complete the above-mentioned data processing method.

[0099] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0100] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0101] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire multiple archive logs corresponding to the first database, where the archive logs are used to record data change operations of the first database; Generate a plurality of preset log files, each of which stores at least one archived log; For each of the preset log files, a transaction identifier of a data operation transaction corresponding to each of the archive logs in the preset log file is obtained, wherein the data operation transaction is a logical work unit of a group of data change operations; wherein, for each of the data operation transactions, the data operation transaction corresponds to multiple archive logs, and the multiple archive logs corresponding to the data operation transaction are from the same or different preset log files; Performing multi-threaded parallel scanning and parsing on the plurality of preset log files to obtain the parsing result of each archived log; According to the transaction identifier and parsing result of each archive log, each data operation transaction is sent to a target terminal respectively. The target terminal is used to perform the data change operation on a second database according to the data operation transaction. The second database is a database for data synchronization with the first database.

2. The method according to claim 1, characterized in that The analysis result includes the log type of each archived log; the multi-threaded parallel scanning and analysis of the plurality of preset log files includes: Starting a preset number of log parsers, wherein the preset number of log parsers corresponds one-to-one to the preset number of preset log files; For each preset log file in the preset number of preset log files, scanning each archived log in the preset log file line by line by using a target log parser corresponding to the preset log file; For the first archive log currently scanned, after parsing the first archive log by the target log parser, the log type of the first archive log is obtained; The sending each of the data operation transactions to the target terminal according to the transaction identifier and the parsing result of each of the archive logs comprises: For the first archive log currently scanned in each of the archive logs, according to the transaction identifier and log type of the first archive log, the data change operation corresponding to the first archive log is stored in the target cache space corresponding to the target transaction identifier; wherein the target transaction identifier is the transaction identifier of the data operation transaction corresponding to the first archive log; In a case where all data change operations under the data operation transaction corresponding to the target transaction identifier are stored in the target cache space, the data operation transaction corresponding to the target cache space is sent to the target terminal.

3. The method according to claim 2, characterized in that The log type includes a transaction commit log, and the transaction commit log is used to indicate the end of the corresponding data operation transaction; The sending of the data operation transaction corresponding to the target cache space to the target terminal comprises: In a case where the log type of the first archive log is the transaction commit log, determining whether the remaining preset log files that have not been scanned completely except the preset log file include the second archive log corresponding to the target transaction identifier; When it is determined that the second archive log is included in the remaining preset log files that have not been scanned except the preset log file, after storing the data change operation corresponding to the second archive log in the target cache space corresponding to the target transaction identifier, the data operation transaction corresponding to the first archive log is sent to the target terminal; When it is determined that the second archive log is not included in the remaining preset log files that have not been scanned except the preset log file, sending the data operation transaction corresponding to the first archive log to the target terminal; The second archive log is an archive log corresponding to the target transaction identifier and other than the first archive log that has not been scanned completely.

4. The method according to claim 3, characterized in that The log type also includes a transaction start log or a data modification log, wherein the transaction start log indicates the start of a corresponding data operation transaction, and the data modification log indicates the data change operation of the corresponding data operation transaction; The storing, for each of the archived logs, a first archived log currently scanned, according to a transaction identifier and a log type of the first archived log, a data change operation corresponding to the first archived log to a target cache space corresponding to a target transaction identifier includes: Determine whether a target cache space corresponding to the target transaction identifier has been created in a preset cache space; In the case where the target cache space is not created in the preset cache space, if the log type of the first archive log is the transaction start log, the target cache space is created in the preset cache space; if the log type of the first archive log is the data modification log, after the target cache space is created in the preset cache space, the data change operation recorded in the first archive log is stored in the target cache space; if the log type of the first archive log is the transaction commit log, the target cache space is created in the preset cache space; In the case that the target cache space has been created in the preset cache space, if the log type of the first archive log is the data modification log, the data change operation recorded in the first archive log is stored in the target cache space; if the log type of the first archive log is the transaction commit log, it is determined whether all data change operations under the data operation transaction corresponding to the target transaction identifier are stored in the target cache space, and if it is determined that all data change operations under the data operation transaction corresponding to the target transaction identifier are stored in the target cache space, the data operation transaction corresponding to the target cache space is sent to the target terminal.

5. The method according to claim 3, characterized in that: Each of the preset log files is configured with a corresponding Bloom filter, and the Bloom filter is used to characterize a set of transaction identifiers of archived logs in the corresponding preset log file; the method further includes: For each of the preset log files, recording the transaction identifier of the data operation transaction corresponding to each of the archive logs in the preset log file to the Bloom filter corresponding to the preset log file; In the case where the log type of the first archive log is the transaction commit log, determining whether the remaining preset log files that have not been scanned completely except the preset log file contain the second archive log corresponding to the target transaction identifier includes: In the case where it is determined that the target transaction identifier is located in the target Bloom filter, determining that the remaining preset log files that have not been scanned completely except the preset log file include the second archive log; The target Bloom filter is a Bloom filter corresponding to other preset log files that have not been scanned except the preset log file.

6. The method according to claim 5, characterized in that Before recording the transaction identifier of the data operation transaction corresponding to each archive log in the preset log file to the Bloom filter corresponding to the preset log file, the method further includes: Obtaining the data synchronization delay time between the first database and the second database at the current moment, and the scanning time of the archive log record corresponding to the latest scanned data operation transaction in history, and obtaining at least one historical log file corresponding to the preset log file, wherein the historical log file is a preset log file scanned within a preset historical time period; According to the data synchronization delay time, the scanning time and the at least one historical log file, parameters of the Bloom filter corresponding to the preset log file are configured to determine the capacity corresponding to the Bloom filter and the number of hash functions.

7. The method according to claim 6, characterized in that The configuring parameters of the Bloom filter corresponding to the preset log file according to the data synchronization delay time, the scanning time and the at least one historical log file includes: Obtain the number of times each historical log file scanned within a preset historical time period has been merged, scanned and parsed and the number of transactions contained in each file; Determine the estimated number of times the preset log file is subjected to merge scanning and parsing according to the number of times each historical log file is subjected to merge scanning and parsing; Determine the estimated tolerable false alarm rate corresponding to the preset log file according to the data synchronization delay time, the scanning time, the estimated number of times and a preset maximum tolerable synchronization delay threshold; Determine a predicted value of the number of transactions corresponding to the preset log file according to the number of transactions respectively contained in each historical log file; Determining the capacity of the Bloom filter and the number of hash functions corresponding to the preset log file according to the estimated tolerable false alarm rate and the predicted value of the number of transactions; According to the capacity and the number of the hash functions, parameters of the Bloom filter corresponding to the preset log file are configured.

8. A data processing device, characterized in that: The device comprises: A first acquisition module, used to acquire a plurality of archive logs corresponding to a first database, wherein the archive logs are used to record data change operations of the first database; A file generation module, used to generate a plurality of preset log files, each of which stores at least one archived log; A second acquisition module is used to acquire, for each of the preset log files, a transaction identifier of a data operation transaction corresponding to each of the archive logs in the preset log file, wherein the data operation transaction is a logical work unit of a group of data change operations; wherein, for each of the data operation transactions, the data operation transaction corresponds to multiple archive logs, and the multiple archive logs corresponding to the data operation transaction are from the same or different preset log files; A log parsing module, used to perform multi-threaded parallel scanning and parsing on a plurality of the preset log files, and obtain the parsing result of each archived log; A data synchronization module is used to send each of the data operation transactions to a target terminal according to the transaction identifier and parsing result of each archive log. The target terminal is used to perform the data change operation on a second database according to the data operation transaction. The second database is a database that performs data synchronization processing with the first database.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 7.

Images