Log analysis method, system and equipment of database and readable storage medium
By parsing the Oracle database log collection and dynamically selecting checkpoints to resolve abnormal logs, the efficiency of data synchronization and system robustness are improved, solving the problem of low synchronization efficiency in existing technologies.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-22
- Publication Date
- 2026-03-31
AI Technical Summary
In existing technologies, parsing anomalies during incremental data synchronization in Oracle databases leads to redundant processing and low synchronization efficiency, especially when encountering abnormal logs, which require reading the logs from the beginning, resulting in low synchronization efficiency.
By parsing the log set, the log with the earliest generation time is determined as the log to be parsed. If there are no abnormal logs, a checkpoint is generated based on the log with the latest generation time, and the log is marked as a parsed log. If there are abnormal logs, the target checkpoint is determined from the generated checkpoints according to the error type of the abnormal logs, and a new log set is dynamically selected for parsing.
When parsing anomalies are encountered during data synchronization, rollback is performed by dynamically selecting checkpoints, which reduces the load on the source database and improves synchronization efficiency and system robustness.
Smart Images
Figure CN121765014A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of database technology, specifically to a database log parsing method, system, device, and computer-readable storage medium. Background Technology
[0002] The need for incremental data synchronization in Oracle databases is becoming increasingly urgent, requiring the use of log parsing tools like LogMiner to mine logs and ensure data consistency between the source and destination databases.
[0003] In related technologies, the starting point for log mining is often determined based on the source database system change number (SCN). The synchronization process relies on the log itself for location. When faced with parsing anomalies, this type of solution needs to backtrack to the earliest active transaction SCN to read the log from the beginning, resulting in redundant processing and low synchronization efficiency. Summary of the Invention
[0004] This application provides a database log parsing method, system, device, and computer-readable storage medium, which can solve the technical problems of repetitive processing and low synchronization efficiency when parsing anomalies exist in the prior art.
[0005] In a first aspect, embodiments of this application provide a database log parsing method, the database log parsing method comprising: The log set is parsed, which includes n consecutively generated logs. The log with the earliest generation time in the log set is the log with the earliest generation time in the logs to be parsed, and n is a first preset value. If there are no abnormal logs in the log set, a checkpoint is generated based on the log with the latest generation time in the log set, wherein the checkpoint contains the identity identifier of the log with the latest generation time in the log set; The logs included in the log set are marked as parsed logs. A new log set is determined from the logs to be parsed. The new log set is used as the log set, and the log set is parsed. If there are abnormal logs in the log set, the target checkpoint is determined from the generated checkpoints according to the error type of the abnormal logs. A new log set is obtained based on the target log and the error type of the abnormal log. The new log set is used as the log set and parsed. The identity identifier of the target log is the identity identifier contained in the target checkpoint.
[0006] In conjunction with the first aspect, in one implementation, determining the target checkpoint from the generated checkpoints based on the error type of the exception log includes: If the error type of the abnormal log is not supported by the log type, then the checkpoint corresponding to the log set to which the root log of the abnormal log belongs is determined as the target checkpoint. If the error type is a connection failure, then the checkpoint with the latest generation time is determined as the target checkpoint.
[0007] In conjunction with the first aspect, in one implementation, the checkpoint further includes transaction status information, and determining the target checkpoint from the generated checkpoints based on the error type of the exception log includes: If the error type is "incomplete large object read", then the checkpoint that contains the identity identifier of the large object in the transaction status information and has the earliest generation time is determined as the target checkpoint.
[0008] In conjunction with the first aspect, in one implementation, obtaining a new log set based on the target log and the error type of the exception log includes: If the error type of the abnormal log is log type not supported or connection failure, then the logs generated within the preset time period are used as a new log set, wherein the start time of the preset time period is the generation time of the target log, and the end time of the preset time period is the generation time of the abnormal log. If the error type of the abnormal log is incomplete reading of a large object, then the logs generated within the preset time period and the m logs generated after the abnormal log are used as a new log set, where the start time of the preset time period is the generation time of the target log, the end time of the preset time period is the generation time of the abnormal log, and m is a second preset value.
[0009] In conjunction with the first aspect, in one implementation, after the log generation checkpoint based on the latest generation time in the log set, the method further includes: If the number of generated checkpoints is greater than K, then the checkpoint with the earliest generation time among the generated checkpoints will be removed, where K≥2.
[0010] Secondly, embodiments of this application provide a database log parsing system, characterized in that the database log parsing system includes: The parsing module is used to parse the log set, which includes n consecutively generated logs. The log with the earliest generation time in the log set is the log with the earliest generation time in the log to be parsed, and n is a first preset value. The generation module is used to generate a checkpoint based on the log with the latest generation time in the log set if there are no abnormal logs in the log set. The checkpoint contains the identity identifier of the log with the latest generation time in the log set. The first determining module is used to mark the logs included in the log set as parsed logs, determine a new log set from the logs to be parsed, and use the new log set as the log set to parse the log set; The second determining module is used to determine the target checkpoint from the generated checkpoints based on the error type of the abnormal log if there is an abnormal log in the log set. The module is used to obtain a new log set based on the target log and the error type of the abnormal log, and to parse the log set using the new log set as the log set. The identity identifier of the target log is the identity identifier contained in the target checkpoint.
[0011] In conjunction with the second aspect, in one implementation, the second determining module is specifically used for: If the error type of the abnormal log is not supported by the log type, then the checkpoint corresponding to the log set to which the root log of the abnormal log belongs is determined as the target checkpoint. If the error type is a connection failure, then the checkpoint with the latest generation time is determined as the target checkpoint.
[0012] In conjunction with the second aspect, in one implementation, the second determining module is further configured to: If the error type is "incomplete large object read", then the checkpoint that contains the identity identifier of the large object in the transaction status information and has the earliest generation time is determined as the target checkpoint.
[0013] Thirdly, embodiments of this application provide a database log parsing device, the database log parsing device including a processor, a memory, and a database log parsing program stored in the memory and executable by the processor, wherein when the database log parsing program is executed by the processor, it implements the steps of the database log parsing method as described in the first aspect.
[0014] Fourthly, embodiments of this application provide a computer-readable storage medium, characterized in that the computer-readable storage medium stores a database log parsing program, wherein when the database log parsing program is executed by a processor, it implements the steps of the database log parsing method as described in the first aspect.
[0015] The beneficial effects of the technical solutions provided in this application include: The log set is parsed, which includes n consecutively generated logs. The earliest generated log in the log set is the earliest generated log in the logs to be parsed, and n is a first preset value. If there are no abnormal logs in the log set, a checkpoint is generated based on the latest generated log in the log set. The checkpoint contains the identity identifier of the latest generated log in the log set. The logs included in the log set are marked as parsed logs. A new log set is determined from the logs to be parsed. The new log set is used as the log set for parsing. If there are abnormal logs in the log set, a target checkpoint is determined from the generated checkpoints according to the error type of the abnormal log. A new log set is obtained based on the target log and the error type of the abnormal log. The new log set is used as the log set for parsing. The identity identifier of the target log is the identity identifier contained in the target checkpoint. By generating and dynamically selecting checkpoints, different checkpoints can be selected for rollback when parsing anomalies are encountered during data synchronization, reducing the load on the source database and improving synchronization efficiency and system robustness. Attached Figure Description
[0016] Figure 1 This is a flowchart illustrating an embodiment of the log parsing method for the database in this application; Figure 2 This is a diagram illustrating the log parsing process; Figure 3 A schematic diagram illustrating the method for determining target checkpoints; Figure 4 This is a schematic diagram of the architecture of an embodiment of the log parsing system for the database of this application; Figure 5 This is a schematic diagram of the hardware structure of the database log parsing device involved in the embodiments of this application. Detailed Implementation
[0017] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.
[0018] First, some of the technical terms used in this application will be explained to help those skilled in the art understand this application.
[0019] Large objects: These are data types used in databases to store extremely large amounts of data. To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0020] In a first aspect, embodiments of this application provide a method for parsing database logs.
[0021] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the log parsing method for the database in this application. Figure 1 As shown, the database log parsing methods include: Step S10: Parse the log set, which includes n consecutively generated logs. The log set with the earliest generation time is the log with the earliest generation time in the log set to be parsed, and n is a first preset value. In one embodiment, reference is made to Figure 2 , Figure 2 This is a diagram illustrating the log parsing process, such as... Figure 2 As shown, assuming the Logminer has already parsed the redo log with SCN=999, and the total range of logs to be parsed is consecutive redo logs with SCN from 1000 to 5000, and the first preset value n is set to 500, then the current log set must contain 500 consecutively generated logs, and the log with the earliest generation time in the set is the log with the earliest generation time in the logs to be parsed (i.e., the log with SCN=1000). Therefore, the log set to be parsed this time is specifically 500 consecutive logs with SCN=1000 to 1499. During the parsing process, the operation type (such as INSERT / UPDATE), associated table name, transaction ID, and other core information of each log can be obtained.
[0022] Step S20: If there are no abnormal logs in the log set, a checkpoint is generated based on the log with the latest generation time in the log set, wherein the checkpoint contains the identity identifier of the log with the latest generation time in the log set. In one embodiment, such as Figure 2 As shown, if no abnormal logs such as log type incompatibility or connection failure are detected after parsing the log set from SCN=1000 to 1499, then a checkpoint is generated based on the log with the latest generation time in the set (i.e., the log with SCN=1499). This checkpoint is named Checkpoint_1, and its core identity identifier is SCN=1499.
[0023] Step S30: Mark the logs included in the log set as parsed logs, determine a new log set from the logs to be parsed, use the new log set as the log set, and parse the log set. In one embodiment, after the generation of Checkpoint_1 is completed, 500 logs with SCN=1000 to 1499 are marked as parsed. At this time, the range of logs to be parsed is updated to SCN=1500 to 5000. According to the rule of n=500, the new log set is determined to be the continuous logs with SCN=1500 to 1999. Then, the parsing operation of step S10 is performed with this new set as the object.
[0024] Step S40: If there are abnormal logs in the log set, then determine the target checkpoint from the generated checkpoints according to the error type of the abnormal logs; In one embodiment, if there are abnormal logs when parsing the log set, the previously generated checkpoints are loaded, and the target checkpoint is determined based on the error type reported by the abnormal logs.
[0025] Step S50: Obtain a new log set based on the target log and the error type of the abnormal log. Use the new log set as the log set and parse the log set. The identity identifier of the target log is the identity identifier contained in the target checkpoint.
[0026] In one embodiment, based on the target checkpoint obtained in step S40, which marks the latest generated log, the target log is used as the starting point for rollback and rereading. Then, based on the error type of the abnormal log, the endpoint after rollback and rereading is determined, thereby obtaining a new log set. The new log set is then used as the log set for parsing.
[0027] In this embodiment, a log set is parsed, comprising n consecutively generated logs, where the earliest generated log in the log set is the earliest generated log in the logs to be parsed, and n is a first preset value. If no abnormal logs exist in the log set, a checkpoint is generated based on the latest generated log in the log set, wherein the checkpoint contains the identity identifier of the latest generated log in the log set. The logs included in the log set are marked as parsed logs, and a new log set is determined from the logs to be parsed. The new log set is used as the log set for parsing. If abnormal logs exist in the log set, a target checkpoint is determined from the generated checkpoints based on the error type of the abnormal logs. A new log set is obtained based on the target log and the error type of the abnormal logs. The new log set is used as the log set for parsing, wherein the identity identifier of the target log is the identity identifier contained in the target checkpoint. By generating and dynamically selecting checkpoints, different checkpoints can be selected for rollback when parsing anomalies are encountered during data synchronization, reducing the load on the source database and improving synchronization efficiency and system robustness.
[0028] Further, in one embodiment, determining the target checkpoint from the generated checkpoints based on the error type of the exception log includes: Step S401: If the error type of the abnormal log is "log type not supported", then the checkpoint corresponding to the log set to which the root log of the abnormal log belongs is determined as the target checkpoint. In one embodiment, such as Figure 3 As shown, Figure 3 This diagram illustrates the method for determining the target checkpoint. Assume that log A has an SCN of 1300 and log B has an SCN of 1600. The content of log B depends on log A. When parsing the set of SCNs from 1500 to 1999, log A has already been successfully mined independently. However, log B does not include log A during mining, so it cannot be mined normally and will be displayed as an unsupported log. Using this as a marker, we roll back and reread, including the range of log A within the mining range of log B, so that log B can be obtained normally. That is, tracing back to the earliest dependent log A of log B, it is SCN=1300. The log set to which SCN=1300 belongs corresponds to the already generated Checkpoint_1. Therefore, Checkpoint_1 is determined as the target checkpoint. Step S402: If the error type is a connection failure, then the checkpoint with the latest generation time is determined as the target checkpoint.
[0029] In one embodiment, such as Figure 3 As shown, if the connection between the database and the parsing tool is interrupted (connection failure type abnormal) when parsing the set of SCN=2000 to 2499, and the checkpoints that have been generated at this time are Checkpoint_1 (SCN=1499) and Checkpoint_2 (SCN=1999), then the Checkpoint_2 with the latest generation time is selected as the target checkpoint.
[0030] Furthermore, in one embodiment, the checkpoint further includes transaction status information, and determining the target checkpoint from the generated checkpoints based on the error type of the exception log includes: Step S403: If the error type is incomplete reading of a large object, then the checkpoint that contains the identity identifier of the large object in the transaction status information and has the earliest generation time is determined as the target checkpoint.
[0031] In one embodiment, such as Figure 3As shown, the checkpoint also includes transaction status information. When parsing the log set from SCN=2500 to 2999, the log with SCN=2700 was detected as a large object shard data log. The unique identifier of the associated large object is LOB_ID=1001. After parsing, it was found that only 3 shards of this large object were captured (out of a total of 5 shards), triggering a large object incomplete read exception. Transaction status information is extracted from the generated Checkpoint_1, Checkpoint_2, and Checkpoint_3, and checkpoints containing LOB_ID=1001 are selected. Assuming that the transaction status information of Checkpoint_2 contains transaction ID=10... Operation log 6 clearly indicates "Associated large object ID=1001, transaction start SCN=1500"; Checkpoint_3 transaction status information contains subsequent operation records for transaction ID=106, which are also associated with LOB_ID=1001; therefore, among Checkpoint_2 and Checkpoint_3 containing LOB_ID=1001, the Checkpoint_2 with the earliest generation time is selected and determined as the target checkpoint, because Checkpoint_2 records the initial association information of the large object operation, and rolling back to this checkpoint can recapture the complete log chain from the beginning stage of the large object operation.
[0032] Furthermore, in one embodiment, obtaining a new log set based on the target log and the error type of the abnormal log includes: Step S501: If the error type of the abnormal log is log type not supported or connection failure, then the logs generated within the preset time period are used as a new log set, wherein the start time of the preset time period is the generation time of the target log, and the end time of the preset time period is the generation time of the abnormal log. In one embodiment, if the error type of the abnormal log is not incomplete reading of a large object, such as log type not supported or connection failure, the target checkpoint_1 contains an identity identifier of SCN=1499 (i.e., the target log is the log with SCN=1499), the abnormal log is SCN=1700, the start time of the preset time period is the generation time of SCN=1499, and the end time is the generation time of SCN=1700. Therefore, the new log set is the continuous logs from SCN=1500 to 1700 within this time period. The log mining session is reconstructed, and this set is used as the object for re-parsing.
[0033] Step S502: If the error type of the abnormal log is incomplete reading of a large object, then the logs generated within the preset time period and the m logs generated after the abnormal log are taken as a new log set, wherein the start time of the preset time period is the generation time of the target log, the end time of the preset time period is the generation time of the abnormal log, and m is a second preset value.
[0034] In one embodiment, if the abnormal log is SCN=2200 (missing large object fragments leading to incomplete reading), the target checkpoint is the earliest checkpoint Checkpoint_2 (SCN=1999) containing the large object ID. The preset time period is the generation period from SCN=1999 to 2200, and the second preset value m=100. Therefore, the new log set is the logs from SCN=2000 to the abnormal log SCN=2200, plus 100 subsequent logs from SCN=2201 to 2300. The log mining session is reconstructed and parsed again. If there is still an error of incomplete large object reading at SCN=2300, the process returns to the target log SCN=2000 and continues to extend the endpoint until all fragment logs of the large object are completely captured.
[0035] Furthermore, in one embodiment, after the checkpoint based on the latest log generation time in the log set, the method further includes: If the number of generated checkpoints is greater than K, then the checkpoint with the earliest generation time among the generated checkpoints will be removed, where K≥2.
[0036] In one embodiment, considering the need to balance rollback fault tolerance with storage resource consumption, a preset threshold K≥2 is used. In this embodiment, considering the log parsing batch size (n=500) and the frequency of abnormal rollback, K=3 is set, meaning that the checkpoint set retains a maximum of 3 of the most recent checkpoints. The specific execution scenario is as follows: Assuming the generated checkpoints are Checkpoint_1 (SCN=1499), Checkpoint_2 (SCN=1999), and Checkpoint_3 (SCN=2499), the total number of checkpoints is 3. After parsing the log set from SCN=2500 to 2999 (without anomalies), Checkpoint_4 is generated based on the log with SCN=2999. Checkpoint_1, which was generated earliest, is selected from the checkpoint set and removed from the set. Finally, the checkpoint set retains the three most recent checkpoints: Checkpoint_2, Checkpoint_3, and Checkpoint_4.
[0037] In this embodiment, by retaining the latest K checkpoints, the rollback requirements in most abnormal scenarios can be met, and the waste of storage resources caused by the infinite growth of the number of checkpoints can be avoided. At the same time, removing the earliest checkpoint will not affect the rollback effect, and subsequent abnormal rollbacks do not need to rely on outdated checkpoints.
[0038] In embodiments of this application, it further includes: Initialize LogMiner, query existing checkpoints, and if this is the first time LogMiner is started, obtain the current starting SCN, create and save two identical checkpoints (Checkpoint_0, Checkpoint_0). If this is not the first startup: Load the two latest consecutive checkpoints that have been saved (Checkpoint_N-1 and Checkpoint_N).
[0039] Secondly, embodiments of this application also provide a database log parsing system.
[0040] In one embodiment, reference is made to Figure 4 , Figure 4 This is a schematic diagram of the architecture of an embodiment of the log parsing system for the database of this application. Figure 4 As shown, the database log parsing system includes: Parsing module 10 is used to parse a log set, which includes n consecutively generated logs. The log set with the earliest generation time is the log set with the earliest generation time to be parsed, and n is a first preset value. The generation module 20 is used to generate a checkpoint based on the log with the latest generation time in the log set if there are no abnormal logs in the log set. The checkpoint includes the identity identifier of the log with the latest generation time in the log set. The first determining module 30 is used to mark the logs included in the log set as parsed logs, determine a new log set from the logs to be parsed, and use the new log set as the log set to parse the log set; The second determining module 40 is used to determine the target checkpoint from the generated checkpoints according to the error type of the abnormal log if there is an abnormal log in the log set. The module 50 is used to obtain a new log set based on the target log and the error type of the abnormal log, and to parse the log set using the new log set as the log set. The identity identifier of the target log is the identity identifier contained in the target checkpoint.
[0041] Furthermore, in one embodiment, the second determining module 40 is used to: If the error type of the abnormal log is not supported by the log type, then the checkpoint corresponding to the log set to which the root log of the abnormal log belongs is determined as the target checkpoint. If the error type is a connection failure, then the checkpoint with the latest generation time is determined as the target checkpoint.
[0042] Furthermore, in one embodiment, the second determining module 40 is used to: The checkpoint also includes transaction status information. If the error type is incomplete reading of a large object, the checkpoint that contains the identity of the large object in the transaction status information and has the earliest generation time is determined as the target checkpoint.
[0043] Furthermore, in one embodiment, the obtaining module 50 is used for: If the error type of the abnormal log is log type not supported or connection failure, then the logs generated within the preset time period are used as a new log set, wherein the start time of the preset time period is the generation time of the target log, and the end time of the preset time period is the generation time of the abnormal log. If the error type of the abnormal log is incomplete reading of a large object, then the logs generated within the preset time period and the m logs generated after the abnormal log are used as a new log set, where the start time of the preset time period is the generation time of the target log, the end time of the preset time period is the generation time of the abnormal log, and m is a second preset value.
[0044] Furthermore, in one embodiment, the database log parsing system further includes a culling module, used for: If the number of generated checkpoints is greater than K, then the checkpoint with the earliest generation time among the generated checkpoints will be removed, where K≥2.
[0045] The functions of each module in the log parsing system of the above-mentioned database correspond to the steps in the log parsing method embodiment of the above-mentioned database, and their functions and implementation processes will not be described in detail here.
[0046] Thirdly, embodiments of this application provide a database log parsing device, which can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.
[0047] Reference Figure 5 , Figure 5 This is a schematic diagram of the hardware structure of the database log parsing device involved in the embodiments of this application. In this embodiment, the database log parsing device may include a processor, a memory, a communication interface, and a communication bus.
[0048] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.
[0049] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting internal components of the database log parsing device, as well as interfaces used for interconnecting the database log parsing device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.
[0050] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.
[0051] The processor can be a general-purpose processor, which can call the database log parsing program stored in memory and execute the database log parsing method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the database log parsing program is called can be referred to in the various embodiments of the database log parsing method of this application, and will not be repeated here.
[0052] Those skilled in the art will understand that Figure 5 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0053] Fourthly, embodiments of this application also provide a computer-readable storage medium.
[0054] The present application provides a computer-readable storage medium storing a database log parsing program, wherein when the database log parsing program is executed by a processor, it implements the steps of the database log parsing method described above.
[0055] The method implemented when the database log parsing program is executed can be referred to in various embodiments of the database log parsing method of this application, and will not be repeated here.
[0056] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0057] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.
[0058] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.
[0059] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.
[0060] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.
[0061] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.
[0062] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A log analysis method of a database, characterized by, The log parsing method of the database comprises: parsing a log set, the log set comprising n continuously generated logs, the earliest generated log in the log set being the earliest generated log in a to-be-parsed log, n being a first preset value; if there is no abnormal log in the log set, generating a checkpoint based on the latest generated log in the log set, wherein the checkpoint comprises an identity of the latest generated log in the log set; marking the logs included in the log set as parsed logs, determining a new log set from the to-be-parsed logs, taking the new log set as the log set, and parsing the log set; if there is an abnormal log in the log set, determining a target checkpoint from the generated checkpoints according to an error type of the abnormal log; obtaining a new log set according to a target log and the error type of the abnormal log, taking the new log set as the log set, and parsing the log set, wherein the identity of the target log is an identity included in the target checkpoint.
2. The log parsing method of a database according to claim 1, wherein, The method of determining the target checkpoint from the generated checkpoints according to the error type of the abnormal log comprises: if the error type of the abnormal log is that the log type is not supported, determining a checkpoint corresponding to a log set to which a root log of the abnormal log belongs as the target checkpoint; if the error type is a connection failure, determining the latest generated checkpoint as the target checkpoint.
3. The log analysis method of a database according to claim 2, wherein, The checkpoint further comprises transaction state information, and the method of determining the target checkpoint from the generated checkpoints according to the error type of the abnormal log comprises: if the error type is an incomplete large object reading, determining a checkpoint whose identity of a large object is included in the transaction state information and whose generation time is the earliest as the target checkpoint.
4. The log analysis method of a database according to claim 3, wherein, The method of obtaining the new log set according to the target log and the error type of the abnormal log comprises: if the error type of the abnormal log is that the log type is not supported or the connection failure, taking logs generated within a preset time period as the new log set, wherein a start time of the preset time period is the generation time of the target log, and an end time of the preset time period is the generation time of the abnormal log; if the error type of the abnormal log is the incomplete large object reading, taking logs generated within a preset time period and m logs generated after the abnormal log as the new log set, wherein a start time of the preset time period is the generation time of the target log, an end time of the preset time period is the generation time of the abnormal log, and m is a second preset value.
5. The log parsing method of a database according to Claim 1, wherein, The method further comprises: if the number of the generated checkpoints is greater than K, removing a checkpoint whose generation time is the earliest among the generated checkpoints, wherein K≥2.
6. A log analysis system of a database, characterized by, The log parsing system of the database comprises: a parsing module configured to parse a log set, the log set comprising n continuously generated logs, the earliest generated log in the log set being the earliest generated log in a to-be-parsed log, n being a first preset value; The generating module is configured to generate a checkpoint based on a log with the latest generation time in the log set if there is no abnormal log in the log set, wherein the checkpoint contains an identity of the log with the latest generation time in the log set; The first determining module is configured to mark a log included in the log set as a parsed log, determine a new log set from the to-be-parsed logs, and parse the log set by taking the new log set as the log set; The second determining module is configured to determine a target checkpoint from the generated checkpoints according to an error type of the abnormal log if there is the abnormal log in the log set; The obtaining module is configured to obtain a new log set according to a target log and the error type of the abnormal log, and parse the log set by taking the new log set as the log set, wherein an identity of the target log is an identity contained in the target checkpoint.
7. The log parsing system of a database according to claim 1, wherein, The second determining module is specifically configured to: if the error type of the abnormal log is that a log type is not supported, determine a checkpoint corresponding to a log set to which a root log of the abnormal log belongs as the target checkpoint; if the error type is a connection failure, determine a checkpoint with the latest generation time as the target checkpoint.
8. The log parsing system of a database according to claim 1, wherein, The second determining module is further configured to: if the error type is that a large object is not read completely, determine a checkpoint with the earliest generation time as the target checkpoint, wherein the checkpoint contains an identity of the large object in transaction state information.
9. A log analysis apparatus of a database, characterized by, The database log parsing device includes a processor, a memory, and a database log parsing program stored on the memory and executable by the processor, wherein the database log parsing program, when executed by the processor, implements the steps of the database log parsing method according to any one of claims 1 to 5.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a database log parsing program, wherein the database log parsing program, when executed by the processor, implements the steps of the database log parsing method according to any one of claims 1 to 5.