Timestamp-based log collection method, device, equipment and medium

Abstract

The present disclosure relates to a timestamp-based log collection method, device, equipment and medium, the method comprising: reading a timestamp collection starting point added in a configuration file of a collection task; finding a target transaction log file based on the timestamp collection starting point; reading and analyzing the target transaction log file to determine a log sequence number (LSN) set in the configuration file; starting a collection replication stream based on the LSN, and collecting log data based on the replication stream. The present disclosure can determine the corresponding LSN by using the timestamp collection starting point, greatly reducing the difficulty of obtaining the LSN, reducing the professional requirements for users, and thus facilitating the execution of real-time collection tasks.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a timestamp-based log collection method, apparatus, device, and medium. Background Technology

[0002] Currently, flink-cdc-postgresql (a Flink-based data change collection plugin) does not support setting the collection start point. According to the PostgreSQL source code, when constructing the replication stream object, only setting the LSN (Log Sequence Number) value as the collection start point is supported.

[0003] However, users often cannot obtain the LSN value of the data collection starting point. For real-time data collection tasks, checkpointing must be enabled for users to obtain the LSN value; otherwise, users have no way to obtain it. This places excessive demands on the user's expertise and makes real-time data collection too difficult to use. Summary of the Invention

[0004] To address the aforementioned technical issues, this disclosure provides a timestamp-based log collection method, apparatus, device, and medium.

[0005] According to one aspect of this disclosure, a timestamp-based log collection method is provided, comprising:

[0006] Read the timestamp collection start point added in the configuration file of the data collection task;

[0007] The target transaction log file is located based on the timestamp collection start point;

[0008] Read and parse the target transaction log file to determine the log sequence number (LSN) set in the configuration file;

[0009] Based on the LSN, a data acquisition replication stream is started, and log data is collected based on the replication stream.

[0010] According to another aspect of this disclosure, a timestamp-based log collection device is provided, comprising:

[0011] The time reading module is used to read the timestamp collection start point added in the configuration file of the collection task;

[0012] The file search module is used to search for the target transaction log file based on the timestamp collection start point.

[0013] The LSN determination module is used to read and parse the target transaction log file to determine the log sequence number (LSN) set in the configuration file.

[0014] The log collection module is used to start a collection replication stream based on the LSN and collect log data based on the replication stream.

[0015] According to another aspect of this disclosure, an electronic device is provided, the electronic device comprising:

[0016] processor;

[0017] Memory used to store the processor's executable instructions;

[0018] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the above method.

[0019] According to another aspect of this disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the above-described method.

[0020] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0021] The timestamp-based log collection method, apparatus, device, and medium provided in this disclosure include: reading the timestamp collection start point added in the configuration file of the collection task; locating the target transaction log file based on the timestamp collection start point; reading and parsing the target transaction log file to determine the log sequence number (LSN) set in the configuration file; starting a collection replication stream based on the LSN; and collecting log data based on the replication stream. This disclosure can determine the corresponding LSN using the timestamp collection start point, greatly reducing the difficulty of obtaining the LSN, lowering the professional requirements for users, and thus facilitating the execution of real-time collection tasks. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0023] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of the timestamp-based log collection method described in an embodiment of this disclosure;

[0025] Figure 2 This is a flowchart of the method for finding the target transaction log file as described in an embodiment of this disclosure;

[0026] Figure 3This is a flowchart of the method for determining LSN as described in the embodiments of this disclosure;

[0027] Figure 4 This is a schematic diagram of the code described in the embodiments of this disclosure;

[0028] Figure 5 This is a schematic diagram of the structure of the timestamp-based log collection device described in an embodiment of this disclosure;

[0029] Figure 6 This is a schematic diagram of the structure of the electronic device described in an embodiment of this disclosure. Detailed Implementation

[0030] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0031] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0032] Currently, while users find it difficult to obtain the LSN value of the collection starting point, they can easily obtain the timestamp value. Based on this, embodiments of this disclosure provide a timestamp-based log collection method, apparatus, device, and medium. This disclosure is based on the PostgreSQL source code. When constructing a LogicalStreamBuilder (logic stream builder) object, an LSN needs to be specified; if no LSN is specified, 0 is used by default. If the user-input parameter is a timestamp instead of an LSN, existing technologies cannot directly use the timestamp. This solution converts the timestamp to an LSN and then constructs the logical stream object based on the LSN. For ease of understanding, embodiments of this disclosure are described in detail below.

[0033] First, let's explain some technical terms.

[0034] The transaction log is a crucial component of a database, storing the history of all changes and operations within the database system to ensure data integrity in the event of failures (such as power outages or other server crashes). In PostgreSQL (hereinafter referred to as PG), the transaction log file is called the Write Ahead Log (WAL). This means that before a data page is written to disk, the corresponding operation log must be written to disk; this is the core principle behind ensuring no data loss in the entire database. PostgreSQL is a feature-rich free software object-relational database management system. PostgreSQL's transaction log is Pgwal.

[0035] For ease of management, PG divides the transaction log file into N data file spaces, each data file is called a WAL segment file, and the default size of each file is 16MB.

[0036] The database operation log (XLOG Record) is a logical concept. It can be understood that every change in a PG corresponds to an operation log entry, and these operation logs are stored in the data file space. PG reads these operation log records to perform operations such as fault recovery.

[0037] LSN, or Log Sequence Number, indicates the position where an operation log record is written to the transaction log. The value of the Log Sequence Number (LSN) is an unsigned 64-bit integer. Within the transaction log, the LSN is monotonically increasing and unique. The LSN in a transaction log file represents the position where an operation log record is written to the transaction log file. The LSN can be understood as the offset of the operation log within the transaction log file. The LSN consists of three parts: the logical file ID, the physical file ID, and the offset within the file. For example, in LSN 1 / 4288E228, 1 is the logical file ID, 42 is the physical file ID, and 88E228 is the offset within the WAL segment file.

[0038] Figure 1 This flowchart illustrates a timestamp-based log collection method provided in this embodiment. This method is applicable to PGWAL data synchronization and collection in frameworks such as Flink-CDC and Chunjun. The method can be executed by a timestamp-based log collection device configured on a terminal, which can be implemented in software and / or hardware. (Refer to...) Figure 1 The timestamp-based log collection method includes the following steps S10 to S40.

[0039] S10. Read the timestamp collection start point added in the configuration file of the collection task.

[0040] In this embodiment, the timestamp collection start point added in the configuration file is obtained by reading the configuration file of the collection task. The specific implementation process of this step can be referred to as follows.

[0041] Read the configuration file for the data acquisition task. Specifically, read the configuration file for the Pgwal data acquisition task. Pgwal is the transaction log for PostgreSQL, a feature-rich free software object-relational database management system.

[0042] This embodiment can determine whether an LSN is set in the configuration file; if an LSN is set, the following step S108 can be executed directly based on the LSN, namely: start the collection replication stream based on the LSN, and collect log data based on the replication stream.

[0043] If the LSN value is not set in the configuration file, check if a timestamp collection start point has been added in the configuration file; if so, obtain the timestamp collection start point.

[0044] In one implementation, if the LSN is not set in the configuration file, the timestamp collection start point can be obtained from the configuration file, and then the timestamp collection start point can be converted into an LSN through step S104 and subsequent steps.

[0045] In another implementation, if the LSN value is not set in the configuration file and it is determined that no timestamp collection start point is added in the configuration file, this embodiment can set the LSN value to 0, and execute the following step S108 based on the LSN value of 0.

[0046] S20. Locate the target transaction log file based on the timestamp collection start point. In PostgreSQL, the transaction log file can be simply referred to as the WAL (Write Ahead Log), meaning that the corresponding operation log must be written to disk before data pages are written to disk.

[0047] like Figure 2 As shown, the process of finding the target transaction log file based on the timestamp collection start point in this embodiment can be referred to as follows.

[0048] S201. Connect to PostgreSQL-JDBC and use an SQL statement to query the first transaction log file after the timestamp collection start point. An example SQL statement is as follows:

[0049] SELECT name FROM pg_ls_waldir()where modification>TO_TIMESTAMP(%d)order by modification asc limit 1;

[0050] showdata_directory;

[0051] The SQL statement above queries the database directory for all transaction log files that meet preset conditions, including that the file's modification time is greater than the timestamp collection start time. In other words, it queries the database directory for all transaction log files whose modification time is greater than the timestamp collection start time. After retrieving these transaction log files, it determines the first transaction log file according to the chronological order of their modification times and obtains the filename of that first transaction log file.

[0052] S202. Use SQL statements to query the data storage path of the transaction log file in PostgreSQL.

[0053] S203. Concatenate the first transaction log file and the transaction log file under the data storage path to form the target transaction log file. Specifically, based on the query results in steps S201 and S202 above, concatenate the data storage path of the first transaction log file and the data storage path of the transaction log file in PostgreSQL to form the complete path of the target transaction log file.

[0054] S30. Read and parse the target transaction log file to determine the LSN set in the configuration file. (Refer to...) Figure 3 This embodiment can be implemented through the following steps.

[0055] S301. Convert the target transaction log file to a string. In practice, first use SQL statements to read the target transaction log file as a binary stream, and then convert the binary stream into a Java string.

[0056] S302. After processing the string by splitting it with a newline character, read the string line by line and take each line of the string as the current line string. Repeatedly execute the following steps S303 to S306.

[0057] S303. Determine whether the current line string matches a preset keyword. The characters following the keyword are characters representing the operation time.

[0058] In a specific implementation, the system iteratively checks whether a preset keyword, such as "commit" or "abort," is matched in the current line of text. The keyword is followed by the operation time. For example... Figure 4The extracted code includes "COMMIT 2023-02-22 10:09:20.684301" and "COMMIT 2023-02-22 10:55:07.539116". COMMIT is the matched keyword, and the string of characters following COMMIT represents the operation time, consisting of year, month, day, hour, minute, and second.

[0059] If no keyword is matched in the current line string, skip the current line string, take the next line string as the new current line string, and execute step S303 again. If a keyword is matched in the current line string, execute step S304 as follows.

[0060] S304. When a keyword is matched in the current line string, extract the operation time after the keyword.

[0061] S305. Compare the operation time with the timestamp collection start point.

[0062] S306. If the operation time is less than or equal to the timestamp collection start point, cache the current line string into a preset variable and replace the original string in the variable.

[0063] In this embodiment, each time a keyword is matched, the operation time following the keyword is extracted; the operation time is compared with the timestamp collection start point; if the operation time is less than or equal to the timestamp collection start point, the current line string is cached in a preset variable, and the original string in the variable is replaced. Then, the next line string is used as the new current line string, and step S303 is executed again; this process continues until the operation time extracted from the current line string is greater than the timestamp collection start point, at which point the loop stops and exits.

[0064] S307. If the operation time is greater than the timestamp collection start point, then parse the LSN in the current line string. The parsed LSN is the LSN set in the configuration file.

[0065] Specifically, the loop breaks when the operation time exceeds the timestamp collection start time. After the loop ends, the current line string is parsed to obtain the keyword "LSN:". The value following the keyword "LSN:" is the LSN value that the user is looking for in the configuration file.

[0066] S40. Start the collection replication stream based on LSN, and collect log data based on the replication stream.

[0067] This embodiment includes: setting the LSN value when starting the replication stream; constructing a LogicalStreamBuilder logical stream object based on the LSN value and starting the PGReplicationStream replication stream. Log data is collected by continuously monitoring the PGReplicationStream replication stream; that is, continuously monitoring the PGReplicationStream replication stream, reading one binary stream at a time, serializing the binary stream into a ChangeLog object, where the ChangeLog object is the Java object corresponding to one log data point, that is, the ChangeLog object is the log data to be collected by the user.

[0068] Based on the above embodiments, this method may further include:

[0069] Determine if an LSN is set in the configuration file of the data acquisition task; if an LSN is set, set the value of the LSN directly when starting the replication stream; construct a LogicalStreamBuilder logical stream object based on the value of the LSN, and start the PGReplicationStream replication stream.

[0070] If LSN is not set, and no timestamp collection start point is added in the configuration file, the value of LSN is set to 0; a LogicalStreamBuilder logical stream object is constructed based on the value of LSN, and the PGReplicationStream replication stream is started.

[0071] In the above embodiments, a timestamp-based data collection start point is added to the configuration file of the data collection task. If the configuration file sets a value for the LSN, this LSN is directly used to construct the LogicalStreamBuilder logical stream object. If no LSN value is set, the LSN can be determined using the timestamp-based data collection start point. Specifically, the target transaction log file is located based on the timestamp-based data collection start point; the target transaction log file is read and parsed to determine the LSN set in the configuration file; and then the LogicalStreamBuilder logical stream object is constructed based on the LSN. Furthermore, if there is no timestamp-based data collection start point in the configuration file, this solution can also set the LSN value to 0, thereby constructing the LogicalStreamBuilder logical stream object, i.e., without setting a data collection start point.

[0072] The methods provided in the above embodiments can be applied to log data synchronization and collection within preset frameworks; these preset frameworks include, but are not limited to, Flink-CDC and Chunjun. Specifically, this embodiment is built based on the startLocation setting when constructing the replication logic flow in the PostgreSQL source code. It can be applied to PgWAL data synchronization and collection within open-source frameworks such as Flink-CDC and Chunjun. It complements and extends the functionality of PgWAL log data collection, is compatible with more practical and reliable collection start point settings, and makes collection tasks more flexible.

[0073] In summary, the timestamp-based log collection method provided in this disclosure reads the timestamp collection start point added to the configuration file of the collection task; locates the target transaction log file based on the timestamp collection start point; reads and parses the target transaction log file to determine the LSN set in the configuration file; starts a collection replication stream based on the LSN; and collects log data based on the replication stream. This technical solution, by adding a timestamp collection start point to the configuration file of the collection task, can determine the corresponding LSN using the timestamp collection start point even when the LSN value is not set in the configuration file or is difficult for the user to obtain. This significantly reduces the difficulty of obtaining the LSN, lowers the professional requirements for users, and thus facilitates the execution of real-time collection tasks.

[0074] Furthermore, this technical solution utilizes easily obtainable timestamps as the starting point for data collection, successfully converting timestamps into corresponding LSNs. This significantly reduces data collection redundancy and waste, avoiding the collection of non-target data. If each collection task starts collecting data from scratch, it greatly wastes system performance. Using timestamp-based filtering improves development efficiency, increases collection flexibility, and saves operating costs. For real-time collection tasks, users do not need to focus too much on the composition of the Pgwal log or the LSN acquisition method when checkpointing is not enabled; collection can be completed without concern for these factors. For real-time supplementary collection tasks, a custom time can be set to complete subsequent data collection.

[0075] Figure 5 This is a schematic diagram of a timestamp-based log collection device provided in an embodiment of this disclosure. This device can be used to implement the aforementioned timestamp-based log collection method. (Refer to...) Figure 5 The timestamp-based log collection device includes the following modules.

[0076] The time reading module 510 is used to read the timestamp collection start point added in the configuration file of the collection task;

[0077] The file search module 520 is used to search for the target transaction log file based on the timestamp collection start point.

[0078] LSN determination module 530 is used to read and parse the target transaction log file to determine the log sequence number (LSN) set in the configuration file;

[0079] The log collection module 540 is used to start a collection replication stream based on the LSN and collect log data based on the replication stream.

[0080] The device provided in this embodiment has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0081] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Figure 6 As shown, the electronic device 600 includes one or more processors 601 and memory 602.

[0082] The processor 601 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 600 to perform desired functions.

[0083] The memory 602 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 601 may execute the program instructions to implement the timestamp-based log collection method of the embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0084] In one example, the electronic device 600 may also include an input device 603 and an output device 604, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0085] In addition, the input device 603 may also include, for example, a keyboard, a mouse, etc.

[0086] The output device 604 can output various information to the outside, including determined distance information, direction information, etc. The output device 604 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0087] Of course, for the sake of simplicity, Figure 6 Only some of the components of the electronic device 600 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 600 may include any other suitable components depending on the specific application.

[0088] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program for executing the above-described timestamp-based log collection method.

[0089] The present disclosure provides a computer program product for a timestamp-based log collection method, apparatus, electronic device, and medium, including a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0090] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0091] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A timestamp-based log collection method, characterized in that, include: Read the timestamp collection start point added in the configuration file of the data collection task; The target transaction log file is located based on the timestamp collection start point; Read and parse the target transaction log file to determine the log sequence number (LSN) set in the configuration file; A data acquisition replication stream is started based on the LSN, and log data is collected based on the replication stream; The process of locating the target transaction log file based on the timestamp collection start point includes: Connect to PostgreSQL JDBC and use SQL statements to query the first transaction log file after the timestamp collection start point; Use an SQL statement to query the data storage path of the transaction log file in PostgreSQL; The target transaction log file is obtained by concatenating the first transaction log file and the transaction log files under the data storage path. The step of starting a data acquisition replication stream based on the LSN and acquiring log data based on the replication stream includes: When starting the replication stream, set the value of the LSN; Construct a LogicalStreamBuilder object based on the value of LSN, and start the PGReplicationStream replication stream; Log data is collected by continuously monitoring the PGReplicationStream replication stream.

2. The method according to claim 1, characterized in that, The timestamp collection start point added to the configuration file of the data acquisition task includes: Read the configuration file for the data collection task; If the LSN value is not set in the configuration file, determine whether a timestamp collection start point has been added to the configuration file; In this case, obtain the timestamp collection start point.

3. The method according to claim 2, characterized in that, The method further includes: If the timestamp collection start point is not added to the configuration file, set the value of the LSN to 0; Construct a LogicalStreamBuilder logical stream object based on the value of the LSN.

4. The method according to claim 1, characterized in that, The step of reading and parsing the target transaction log file to determine the LSN set in the configuration file includes: The target transaction log file is formatted and converted to obtain a string; After processing the string by splitting it with a newline character, the string is read line by line, and each line of the string is used as the current line string. Determine whether a preset keyword is matched in the current line string, where the characters following the keyword are characters representing the operation time; When the keyword is matched in the current line string, the operation time after the keyword is extracted; Compare the operation time with the timestamp collection start point; If the operation time is less than or equal to the timestamp collection start point, then the current line string is cached in a preset variable, and the original string in the variable is replaced; If the operation time is greater than the timestamp collection start point, then the LSN in the current line string is parsed, and the parsed LSN is the LSN set in the configuration file.

5. The method according to claim 1, characterized in that, The method is applied to log data synchronization and collection within a preset framework; the preset framework includes Flink-CDC and Chunjun.

6. A timestamp-based log collection device, characterized in that, include: The time reading module is used to read the timestamp collection start point added in the configuration file of the collection task; The file search module is used to search for the target transaction log file based on the timestamp collection start point. The LSN determination module is used to read and parse the target transaction log file to determine the log sequence number (LSN) set in the configuration file. The log collection module is used to start a collection replication stream based on the LSN and collect log data based on the replication stream; The file search module is also used for: Connect to PostgreSQL JDBC and use SQL statements to query the first transaction log file after the timestamp collection start point; Use an SQL statement to query the data storage path of the transaction log file in PostgreSQL; The target transaction log file is obtained by concatenating the first transaction log file and the transaction log files under the data storage path. The log collection module is also used for: When starting the replication stream, set the value of the LSN; Construct a LogicalStreamBuilder object based on the value of LSN, and start the PGReplicationStream replication stream; Log data is collected by continuously monitoring the PGReplicationStream replication stream.

7. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the method as described in any one of claims 1-5.

Images