Data parsing method in data synchronization process, storage medium and device

Abstract

The application provides a data analysis method in a data synchronization process, a storage medium and equipment. The data analysis method comprises: sequentially analyzing a source end database log of serial records to obtain a single transaction operation; storing the transaction operation in a storage area of a transaction in the memory; detecting a trigger event of checking the transaction in the memory; checking all transactions in the memory; judging whether there is a long transaction whose current life cycle exceeds a preset maximum life cycle, and if so, removing the long transaction from the memory; obtaining a position of the earliest operation in the database log in the memory, and updating the position as a lowest analysis point. The data analysis method can avoid limiting the lowest analysis point at the start operation of the long transaction, thereby helping to reduce the data that needs to be repeatedly analyzed during recovery analysis, and further helping to improve the data processing efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database, in particular to a data parsing method in data synchronization process, a storage medium and equipment. BACKGROUND

[0002] In the process of real-time data synchronization by data synchronization software, it is necessary to analyze database logs to obtain incremental data. For a relational database, a large number of client programs will simultaneously connect to the database to perform data addition, deletion and modification operations. The database log system will serially record all operations on the database in the database log. This results in the database transactions generated by different clients being interlaced in the database log. In the process of parsing the database log, the data synchronization software will store and assemble the parsed single operation in memory according to the transaction, and then upload the complete transaction to the target end. Therefore, if an exception occurs during parsing, the parsed transactions in the memory that have not been uploaded will be lost.

[0003] Therefore, during the parsing of the database log, a parsing minimum point is usually reserved in the database log. The parsing minimum point is the position of the earliest start operation in the transaction that has not been uploaded. When an exception occurs in data synchronization, the parsing can be restarted from the database log position of the parsing minimum point, so as to reacquire all transactions that have not been uploaded. However, when there is a long transaction in the database log, the parsing minimum point will remain at the start operation of the long transaction. This results in the need to repeatedly parse a large amount of data during each recovery and parsing, which affects the processing efficiency of the system. SUMMARY

[0004] An object of the present application is to provide a data parsing method in data synchronization process, a storage medium and equipment capable of solving any of the above problems.

[0005] A further object of the present application is to ensure the timeliness of updating the parsing minimum point.

[0006] In particular, the present application provides a data parsing method in data synchronization process, comprising:

[0007] sequentially parsing the serially recorded source database log to obtain single transaction operations;

[0008] storing the transaction operations in the memory area of the transaction to which the transaction operation belongs, respectively;

[0009] detecting a triggering event for checking the transactions in the memory;

[0010] iteratively checking all transactions in the memory;

[0011] determining whether there is a long transaction whose current life cycle exceeds a preset maximum life cycle, and if so, removing the long transaction from the memory;

[0012] obtaining a position of an earliest operation in the memory in the database log, and updating the position as a parsing bottom.

[0013] Optionally, the step of obtaining the current life cycle of the transaction comprises:

[0014] obtaining a generation time of a start operation of the transaction in the database log, denoted as a first time;

[0015] obtaining a generation time of a parsing position of the database log when the trigger event is detected, denoted as a second time;

[0016] taking a time length from the first time to the second time as the current life cycle.

[0017] Optionally, the trigger event comprises:

[0018] there is a complete transaction in the memory to be transmitted to a target end.

[0019] Optionally, the step of obtaining a single transaction operation comprises:

[0020] obtaining transaction number information of the single transaction operation;

[0021] determining a transaction to which the single transaction operation belongs according to the transaction number information.

[0022] Optionally, the step of removing the long transaction from the memory comprises:

[0023] moving the long transaction to an external storage.

[0024] Optionally, after the step of moving the long transaction to the external storage, comprising:

[0025] after a single transaction operation is subsequently obtained from the database log,

[0026] determining whether the single transaction operation belongs to the long transaction stored in the external storage, and if so, storing the single transaction operation to a corresponding storage position in the external storage; and if not, storing the single transaction operation to a corresponding storage position in the memory.

[0027] Optionally, before resuming parsing of the database log after an exception occurs in data synchronization, comprising:

[0028] scanning the external storage;

[0029] determining whether a long transaction is stored in the external storage, and if so, recording transaction number information of the long transaction in the internal storage.

[0030] Optionally, the step of detecting a triggering event of checking the transactions in the internal storage comprises:

[0031] taking a parsing position of the database log at the time of detecting the triggering event as a commit lower bound.

[0032] According to another aspect of the present application, there is also provided a machine readable storage medium having stored thereon a machine executable program which, when executed by a processor, implements the data parsing method in the data synchronization process in any of the above.

[0033] According to yet another aspect of the present application, there is also provided a computer device comprising a memory, a processor, and a machine executable program stored on the memory and running on the processor, and the processor implements the data parsing method in the data synchronization process in any of the above when executing the machine executable program.

[0034] The data parsing method of the present application detects a triggering event of checking the transactions in the internal storage, checks all the transactions in the internal storage, determines whether there is a long transaction whose current life cycle exceeds a preset maximum life cycle, and if so, removes the long transaction from the internal storage, and acquires a position of the earliest operation in the database log in the internal storage, and updates the position as a parsing lower bound. By checking the transactions in the internal storage, and then removing the transactions whose current life cycle exceeds the preset maximum life cycle from the internal storage, the long transaction can be first removed to a storage position outside the internal storage for storage, so that when a data synchronization exception occurs, the long transaction will not be lost like other transactions in the internal storage. Therefore, when recovering and parsing, the long transaction does not need to be re-parsed, so that the parsing lower bound of the database log can be moved up. Therefore, the parsing lower bound can be prevented from being limited at the start operation of the long transaction, which helps to reduce the data that needs to be repeatedly parsed when recovering and parsing, and thus helps to improve the data processing efficiency. Moreover, the long transaction is stored in other storage positions, which helps to avoid the loss of the front operations of the long transaction when the database log is cleaned.

[0035] Further, the triggering event in the data parsing method of the present application comprises an event of uploading a complete transaction in the internal storage. By taking the uploading of a complete transaction in the internal storage as the triggering event, the checking of the internal storage is more random, which avoids the checking of the internal storage too frequently and affects the parsing efficiency, and also helps to ensure the timeliness of updating the parsing lower bound.

[0036] The above and other objects, advantages and features of the present application will become more apparent from the following detailed description considered in conjunction with the accompanying drawings, in which: BRIEF DESCRIPTION OF DRAWINGS

[0037] Some specific embodiments of the present application will now be described in detail with reference to the following drawings, which are provided by way of explanation of the application and are not limiting of the present application. In the drawings:

[0038] Figure 1 Fig. 1 is a schematic diagram of a database log parsing minimum point varying with parsing;

[0039] Figure 2 Fig. 2 is a schematic diagram of a database log with a long transaction;

[0040] Figure 3 Fig. 3 is a schematic flow chart of a data parsing method according to one embodiment of the present application;

[0041] Figure 4 Fig. 4 is a schematic flow chart of a step of obtaining a single transaction operation in the data parsing method according to one embodiment of the present application;

[0042] Figure 5 Fig. 5 is a schematic flow chart of a step of obtaining a current life cycle of a transaction in the data parsing method according to one embodiment of the present application;

[0043] Figure 6 Fig. 6 is a schematic flow chart of a step of parsing a database log after a long transaction is moved to an external storage in the data parsing method according to one embodiment of the present application;

[0044] Figure 7 Fig. 7 is a schematic flow chart of a step of resuming parsing after an abnormality occurs in a synchronization process in the data parsing method according to one embodiment of the present application;

[0045] Figure 8 Fig. 8 is a schematic flow chart of a data parsing method according to another embodiment of the present application;

[0046] Figure 9 Fig. 9 is a schematic diagram of a machine readable storage medium according to one embodiment of the present application;

[0047] Figure 10 Fig. 10 is a schematic diagram of a computer device according to one embodiment of the present application. DETAILED DESCRIPTION

[0048] Those skilled in the art should understand that the embodiments described below are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. These partial embodiments are intended to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of the present invention.

[0049] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0050] Figure 1 This is a diagram illustrating how the lowest point of a database log parsing changes during parsing. Figure 2 This is a diagram of a database log with long transactions.

[0051] The following reference Figure 1 and Figure 2 The existing data parsing methods are explained.

[0052] Reference Figure 1 As shown, in the database log, operations 1, 3, 7, 10, 12, and 14 represent the same transaction, let's call it transaction A; operations 2, 4, 11, and 15 represent the same transaction, let's call it transaction B; operations 5, 6, 8, and 9 represent the same transaction, let's call it transaction C; and operations 13 and 16 represent the same transaction, let's call it transaction D. In other words, all operations on the database are recorded sequentially in the database log (i.e., operations from different transactions overlap). The first operation (e.g., operation 1 in transaction A, operation 2 in transaction B) indicates the start of the transaction, and the last operation (e.g., operation 14 in transaction A, operation 15 in transaction B) indicates the end of the transaction. The start and commit operations together represent a complete transaction.

[0053] In the prior art, when the synchronization software parses the database log, each time a transaction operation is parsed, the transaction operation is stored in the memory. And, the transaction operations belonging to the same transaction are arranged and assembled, for example, in the memory, the operations 1, 3, 7, 10, 12, and 14 of the transaction A are sequentially arranged and assembled. Each time a complete transaction is assembled, the synchronization software uploads it to the target end. For example, the operations 1, 3, 7, 10, 12, and 14 of the transaction A are all parsed and stored in the memory of the synchronization software, that is, the complete transaction A can be assembled, and the synchronization software uploads the transaction A to the target end.

[0054] Further, in the parsing process, the lowest parsing point and the lowest commit point are kept in the database log. Specifically, when the operation 9 is parsed, in the memory of the synchronization software, there are the operations 1, 3, and 7 of the transaction A, the operations 2 and 4 of the transaction B, and the operations 5, 6, 8, and 9 of the transaction C. Therefore, the transaction C is a complete transaction, and the synchronization software uploads the transaction C. The transactions A and B are not complete, and therefore are not uploaded, that is, the un-uploaded transactions in the memory. At this time, in the database log, the position of the operation 1 of the transaction A is the lowest parsing point, that is, the point P0 of the middle graph of FIG. 6. The position of the operation 9 of the transaction C is the lowest commit point, that is, the point P1 of the middle graph of FIG. 6. Figure 1 Figure 1

[0055] Because if the synchronization abnormally occurs and the memory is emptied, the transactions A and B are not uploaded, and the re-parsing needs to start from the operation 1 of the transaction A, the position of the operation 1 of the transaction A is the lowest parsing point. At the same time, because the transaction C has been uploaded, the repeated parsing needs to be discarded, that is, the transactions uploaded before the lowest commit point need to be discarded.

[0056] Continuing to say, when the operation 14 is parsed, in the memory of the synchronization software, there are the operations 1, 3, 7, 10, 12, and 14 of the transaction A, the operations 2, 4, and 11 of the transaction B, and the operation 13 of the transaction D. At this time, the transaction C has been uploaded, and therefore, there is no transaction C in the memory. Therefore, the transaction A is a complete transaction, and the synchronization software uploads the transaction A. The transactions B and D are not complete, and therefore are not uploaded, that is, the un-uploaded transactions in the memory. At this time, in the database log, the position of the operation 2 of the transaction B is the lowest parsing point, that is, the point P0 of the lowermost graph of FIG. 7. The position of the operation 14 of the transaction A is the lowest commit point, that is, the point P1 of the lowermost graph of FIG. 7. Figure 1 Figure 1

[0057] ​​​​If an exception occurs at this time, the memory is emptied, the B transaction and the D transaction have not been uploaded, and re-parsing needs to start from the 2 operation of the B transaction, so the position of the 2 operation of the B transaction is the lowest point of parsing. Meanwhile, the transactions uploaded before the 14 operation need to be discarded.

[0058] Referring to Figure 2 As shown in the above parsing method in the prior art, if there is a long transaction in the database log, assume that 1 and 1000000000 belong to the same transaction, denoted as an E transaction. Because the E transaction cannot be uploaded for a long time, no matter how many transactions have been uploaded between the 1 operation and the 1000000000 operation, the lowest point of parsing remains at the 1 operation. Therefore, when an exception occurs and the parsing is resumed, it needs to start from 1 each time. Therefore, the later the exception occurs, the more data needs to be parsed repeatedly, that is, the more transactions that have been uploaded need to be parsed repeatedly, resulting in waste of processing performance.

[0059] Figure 3 is a schematic flowchart of a data parsing method according to an embodiment of the present application. In this embodiment, the data parsing method can generally include:

[0060] In step S302, the source end database log of serial records is sequentially parsed to obtain single transaction operations.

[0061] In step S304, the transaction operations are respectively stored in the storage area of the transaction in the memory.

[0062] In step S306, a trigger event of checking the transactions in the memory is detected.

[0063] Specifically, the trigger event includes that a complete transaction in the memory is transmitted to the target end. That is, when a complete transaction in the memory is uploaded, the next step is executed.

[0064] It should be noted that in some other embodiments of the present application, the trigger event can also include that the parsed transaction operations reach a preset number, for example, when 10000 operations are parsed from the 1 operation, it is determined that the preset number is reached, and then 20000 operations are parsed from 10001, it is determined that the preset number is reached.

[0065] Alternatively, the trigger event can also include that the parsing time reaches a preset time length. That is, the next step is executed at a fixed time.

[0066] In step S308, all the transactions in the memory are traversed and checked.

[0067] Step S310, judging whether there is a long transaction whose current life cycle exceeds the preset maximum life cycle, if yes, executing step S312, and then executing step S314, if no, executing step S314.

[0068] The preset maximum life cycle can be set according to actual needs.

[0069] Step S312, moving the long transaction out of the memory.

[0070] Specifically, the long transaction can be moved to the external storage.

[0071] It should be noted that in some other embodiments of the present application, other storage methods can also be used to store the long transaction, so that the external storage is moved out of the memory without being lost in an exception. Alternatively, the parsed long transaction can also be directly uploaded to the target end.

[0072] Step S314, acquiring the position of the earliest operation in the database log in the memory, and updating the position as the parsing lower limit.

[0073] Figure 4 is a schematic flowchart of the step of acquiring a single transaction operation in the data parsing method according to an embodiment of the present application.

[0074] In combination with Figure 4 , specifically, in step S302, the step of acquiring a single transaction operation includes:

[0075] Step S402, acquiring transaction number information of a single transaction operation.

[0076] Step S404, determining a transaction to which the single transaction operation belongs according to the transaction number information.

[0077] Specifically, each transaction operation contains corresponding transaction number information, for example, the A transaction and the B transaction in the foregoing description, so that the transaction operation belongs to which transaction can be known through the transaction number information. Then, the transaction operations of the same transaction can be stored and assembled in order.

[0078] Figure 5 is a schematic flowchart of the step of acquiring a current life cycle of a transaction in the data parsing method according to an embodiment of the present application.

[0079] In combination with Figure 5 , specifically, in step S310, the step of acquiring a current life cycle of a transaction includes:

[0080] Step S502, acquiring a generation time of a start operation of a transaction in a database log, denoted as a first time;

[0081] Step S504, obtain the generation time of the parsing position of the database log when the trigger event is detected, denoted as the second time;

[0082] Step S506, take the time length from the first time to the second time as the current survival period.

[0083] In combination Figure 2 As shown, it is assumed that when parsing to 10000, the trigger event of checking the transaction in the memory is detected. Then, the traversal checking of all transactions in the memory is started. Taking the E transaction as an example, the start operation is 1, and the generation time of the 1 operation in the database log is obtained, denoted as the first time. Then, the generation time of the current parsing position, i.e. the generation time of 10000, is obtained, denoted as the second time. The time length from the first time to the second time is the current survival period of the E transaction.

[0084] In combination Figures 2 to 5 As shown, the parsing method of the embodiment is exemplarily described. Figure 2 In the figure, it is assumed that 1 and 1000000000 belong to the same transaction, denoted as the E transaction. It is assumed that 5, 6, 8 and 10 are a transaction, denoted as the F transaction. When parsing to the 10 operation, the F transaction is uploaded. Then, the trigger event is detected. The checking of all transactions in the memory is performed. Taking the E transaction as an example, the start operation is 1, and the generation time of the 1 operation in the database log is obtained, denoted as the first time. Then, the generation time of the current parsing position, i.e. the generation time of 10, is obtained, denoted as the second time. The time length from the first time to the second time is the current survival period of the E transaction. At this time, the current survival period of the E transaction does not exceed the preset maximum survival period, and the parsing lowest point is the 1 operation.

[0085] When parsing to the 10000 operation, another transaction is uploaded. Then, the trigger event is detected. The checking of all transactions in the memory is performed. Taking the E transaction as an example, the start operation is 1, and the generation time of the 1 operation in the database log is obtained, denoted as the first time. Then, the generation time of the current parsing position, i.e. the generation time of 10000, is obtained, denoted as the second time. The time length from the first time to the second time is the current survival period of the E transaction. At this time, the current survival period of the E transaction exceeds the preset maximum survival period, and therefore, the E transaction is put into the external storage, and the parsing lowest point is updated to the earliest operation of the transaction still stored in the memory. For example, the 9000 operation is the start operation of a transaction, and all operations between the 1 operation and the 9000 operation have been uploaded. Then, after the E transaction is put into the external storage, the parsing lowest point is updated to the 9000 operation.

[0086] In the scheme of the embodiment, by setting a trigger event, and after detecting the trigger event, checking the transactions in the memory, and then storing the transactions whose current life cycle exceeds the preset maximum life cycle into the external storage. That is, the long transaction can be stored in the external storage, so that when the data synchronization exception occurs, the long transaction will not be lost with other transactions in the memory. Therefore, when recovering and analyzing, the long transaction does not need to be reanalyzed, so as to make the minimum point of database log analysis move up. Therefore, it can avoid limiting the minimum point of analysis to the start operation of the long transaction, thereby helping to reduce the data that needs to be repeatedly analyzed during recovery and analysis, and further helping to improve the data processing efficiency. Moreover, the long transaction is stored in the external storage, which helps to avoid the loss of the long transaction operation before the database log is cleaned.

[0087] Further, by uploading the complete transaction in the memory as a trigger event, the work of checking the memory is more random. Avoiding checking the memory too frequently and affecting the analysis efficiency, also helps to ensure the timeliness of updating the minimum point of analysis.

[0088] Figure 6 is a schematic flowchart of the step of analyzing the database log after the long transaction is moved to the external storage in the data analysis method according to an embodiment of the application.

[0089] Referring to Figure 6 , specifically, after the step of moving the long transaction to the external storage, the step of analyzing the database log includes:

[0090] Step S602, obtaining a single transaction operation from the database log.

[0091] Step S604, determining whether the single transaction operation belongs to the long transaction stored in the external storage, if yes, executing step S606, if no, executing step S608.

[0092] Step S606, storing the single transaction operation to the corresponding storage position in the external storage.

[0093] Step S608, storing the single transaction operation to the corresponding storage position in the memory.

[0094] Specifically, after the long transaction is stored in the external storage, in the subsequent process of continuously analyzing the database log, if the analyzed data is the operation of the long transaction in the external storage, the operation is stored in the position of the long transaction in the external storage. If the analyzed data is the operation of the transaction in the memory, the operation is stored in the position of the transaction in the memory.

[0095] Similarly, if the commit operation of the long transaction in the external storage is analyzed, the complete long transaction in the external storage is uploaded.

[0096] Figure 7 FIG. 7 is a schematic flow chart of a step of resuming parsing after an exception in a synchronization process according to an embodiment of the present application.

[0097] Referring to FIG. 7, in particular, the step of resuming parsing after an exception generally includes: Figure 7

[0098] Step S702, scanning the external storage.

[0099] Step S704, determining whether a long transaction is stored in the external storage, if yes, executing step S706, and then executing step S708, if no, executing step S708.

[0100] Step S706, recording the transaction number information of the long transaction into the internal storage.

[0101] Step S708, re-parsing the database log from the lowest point of parsing.

[0102] In particular, when detecting that the data synchronization has an exception, after the exception is recovered, the parsing needs to be re-performed. At this time, the external storage is scanned first, if the long transaction is stored in the external storage, the transaction number information of the long transaction is recorded into the internal storage. Then, the database log is re-parsed from the lowest point of parsing.

[0103] As can be understood by those skilled in the art, because the synchronization has an exception, the information in the internal storage is all lost. Therefore, by re-recording the transaction number information of the long transaction in the external storage into the internal storage before resuming parsing, when the operation belonging to the long transaction is parsed subsequently, the belonging position thereof can be known, and the situation of parsing confusion can be avoided.

[0104] Figure 8 FIG. 8 is a schematic flow chart of a data parsing method according to another embodiment of the present application.

[0105] Referring to FIG. 8, in particular, in this embodiment, the data parsing method generally includes: Figure 8

[0106] Step S802, sequentially parsing the source database log of the serial record to obtain a single transaction operation.

[0107] Step S804, storing the transaction operation into the storage area of the transaction in the internal storage.

[0108] Step S806, detecting a trigger event of checking the transaction in the internal storage.

[0109] Step S808, taking the parsing position of the database log when the trigger event is detected as the lowest point of commitment.

[0110] ​​Step S810, all transactions in the memory are traversed and checked.

[0111] Step S812, it is judged whether there is a long transaction whose current life cycle exceeds the preset maximum life cycle, if yes, step S814 is executed, and then step S816 is executed, if not, step S816 is executed.

[0112] The preset maximum life cycle can be set according to actual needs.

[0113] Step S814, the long transaction is moved to the external storage.

[0114] Step S816, the position of the earliest operation in the database log in the memory is acquired, and the position is updated as the parsing lowest point.

[0115] In combination with Figure 2 , the parsing method of the embodiment is exemplarily described. Figure 2 In the table, it is assumed that 1, 1000000000 belong to the same transaction, which is recorded as E transaction. It is assumed that 5, 6, 8, 10 are a transaction, which is recorded as F transaction. When the 10 operation is parsed, the F transaction is uploaded. Then, the trigger event is detected, and the position of the 10 operation is marked as the commit lowest point.

[0116] When the 10000 operation is parsed, another transaction is uploaded. Then, the trigger event is detected, and the position of the 10000 operation is marked as the commit lowest point.

[0117] In the scheme of the embodiment, by setting the commit lowest point, the transaction that has been uploaded can be marked. In the process of re-parsing the database log after the abnormal recovery, the transaction whose commit operation is before the commit lowest point is the transaction that has been uploaded, so these transactions are discarded.

[0118] The embodiment further provides a machine readable storage medium and a computer device. Figure 9 is a schematic diagram of a machine readable storage medium 10 according to an embodiment of the present application. Figure 10 is a schematic diagram of a computer device 20 according to an embodiment of the present application.

[0119] The machine readable storage medium 10 has a machine executable program 11 stored thereon, and the machine executable program 11 is executed by a processor to implement the processing method of the database shared memory buffer pool in any of the above embodiments.

[0120] The computer device 20 can include a memory 220, a processor 210, and a machine executable program 11 stored on the memory 220 and running on the processor 210, and the processor 210 implements the processing method of the database shared memory buffer pool of any of the above embodiments when executing the machine executable program 11.

[0121] It should be noted that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, and can be embodied in any machine readable storage medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or in conjunction with such an instruction execution system, apparatus or device.

[0122] For the description of the present embodiment, the machine readable storage medium 10 can be any device that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus or device, or in conjunction with such an instruction execution system, apparatus or device. More specific examples (non-exhaustive list) of computer readable medium include the following: electrical connections having one or more wires (electronic devices), portable computer diskette (magnetic devices), random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optic devices, and portable compact disk read only memory (CDROM). In addition, the machine readable storage medium 10 can even be paper or other suitable medium on which the program can be printed, as the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by electronic conversion of the optical scanning into an electronically available form, and then storing the program in computer memory.

[0123] It should be understood that parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above described embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system.

[0124] The computer device 20 can be, for example, a server, a desktop computer, a notebook computer, a tablet computer, or a smart phone. In some examples, the computer device 20 can be a cloud computing node. The computer device 20 can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. Computer device 50 can be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules can be located in both local and remote computer system storage media including memory storage devices.

[0125] The computer device 20 can include a processor 210 adapted to execute instructions stored in memory 220, which provides instructions to operations of the stored instructions during operation. The processor 210 can be a single core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 220 can include random access memory (RAM), read only memory, flash memory, or any other suitable memory systems.

[0126] The processor 210 can be connected through a system interconnect (e.g., PCI, PCI-Express, etc.) to an I / O interface adapted to connect the computer device 20 to one or more I / O devices (input / output devices). The I / O devices can include, for example, a keyboard and a pointing device, which can include a touchpad or a touch screen, and so on. The I / O devices can be built-in components of the computer device 20, or can be devices externally connected to the computer device.

[0127] The processor 210 can also be linked through the system interconnect to a display interface adapted to connect the computer device 20 to a display device. The display device can include a display screen as a built-in component of the computer device 20. The display device can also include a computer monitor, a television, or a projector, and so on, externally connected to the computer device 20. In addition, a network interface controller (NIC) can be adapted to connect the computer device 20 to a network through the system interconnect. In some embodiments, the NIC can use any suitable interface or protocol (such as Internet Small Computer System Interface, etc.) to transfer data. The network can be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN), or the Internet, and so on. Remote devices can be connected to the computer device through the network.

[0128] The flowcharts provided by the embodiments are not intended to indicate that the operations of the methods will be executed in any particular order, or that all of the operations of the methods are included in every case. In addition, the methods can include additional operations. Additional changes can be made to the above-described methods within the scope of the technical ideas provided by the embodiments.

[0129] Thus, those skilled in the art will recognize that the present application has been well- described and clearly demonstrated by way of example, and that many changes and modifications can be made thereto without departing from the spirit and scope of the application. It is therefore intended to cover in the appended claims all such changes and modifications that are within the scope of this application.

Claims

1.A method for data parsing in a data synchronization process, comprising: sequentially parsing a source database log of serial records to obtain single transaction operations; storing the transaction operations in a memory area of a transaction to which the transaction operations belong; detecting a trigger event of checking the transactions in the memory; checking all transactions in the memory; judging whether there is a long transaction whose current life cycle exceeds a preset maximum life cycle, and if so, removing the long transaction from the memory; obtaining a position of an earliest operation in the database log, and updating the position as a parsing lower limit; the step of obtaining the current life cycle of a transaction comprises: obtaining a generation time of a start operation of the transaction in the database log, denoted as a first time; obtaining a generation time of a parsing position of the database log when the trigger event is detected, denoted as a second time; taking a time length from the first time to the second time as the current life cycle. 2.The method according to claim 1, wherein the trigger event comprises: a complete transaction in the memory is transmitted to a target. 3.The method according to claim 1, wherein the step of obtaining single transaction operations comprises: obtaining transaction number information of the single transaction operation; and determining a transaction to which the single transaction operation belongs according to the transaction number information. 4.The method according to claim 1, wherein the step of removing the long transaction from the memory comprises: moving the long transaction to an external storage. 5.The method according to claim 4, wherein after the step of moving the long transaction to the external storage, comprising: after subsequently obtaining a single transaction operation from the database log, judging whether the single transaction operation belongs to the long transaction stored in the external storage, and if so, storing the single transaction operation to a corresponding storage position in the external storage; and if not, storing the single transaction operation to a corresponding storage position in the memory. 6.The method according to claim 4, wherein before resuming parsing of the database log after an abnormality occurs in data synchronization, comprising: scanning the external storage; judging whether there is a long transaction stored in the external storage, and if so, recording transaction number information of the long transaction in the memory. 7.The method according to claim 1, wherein after the step of detecting the trigger event of checking the transactions in the memory, comprising: taking a parsing position of the database log when the trigger event is detected as a commit lower limit. 8.A machine readable storage medium having stored thereon a machine executable program, the machine executable program, when executed by a processor, implements the method for data parsing in a data synchronization process according to any one of claims 1 to 7. ​ ​ ​ ​ ​ ​ 9. A computer device comprising a memory, a processor, and a machine executable program stored on the memory and running on the processor, and the processor implements the data parsing method in the data synchronization process according to any one of claims 1 to 7 when executing the machine executable program.

Images