Log data reading method and device, equipment, medium and program product

Abstract

The invention provides a log data reading method and device, equipment, a medium and a program product, and relates to the field of big data. Obtaining a mapping relationship between a plurality of timeline identifiers in a database and corresponding start log serial numbers; the timeline identifier is an identifier generated when the main database and the standby database are switched, whether the log data cross different timeline identifiers or not is judged according to the mapping relation and the starting log serial number and the ending log serial number of the log data, and under the condition that the log data cross different timeline identifiers, the log data cross different timeline identifiers. Reconstructing a log reading process according to a start log serial number of a new timeline identifier, wherein the new timeline identifier is different from the plurality of timeline identifiers; according to the method, automatic reconstruction of the log reading process after main and standby switching is achieved, the problems of low efficiency and poor real-time performance caused by manual operation are avoided, and continuity and consistency of data synchronization of the source end and the target end are guaranteed.

Description

Technical Field

[0001] This application relates to the field of big data, and in particular to a log data reading method, apparatus, device, medium, and program product. Background Technology

[0002] In a real-time data synchronization solution based on database write-ahead logs, write-ahead logs are read from the source database in real time, the read log data is written to a local file, the locally stored write-ahead logs are parsed and processed, and the parsed data is written to the target database, thereby achieving data consistency synchronization between the source and target.

[0003] In existing technologies, log reading methods involve creating physical replication slots in the database and using the physical replication slot interface provided by the JDBC driver to retrieve logs in real time. During master-slave failover, the database undergoes a restart, directly causing the connection between the synchronization tool and the database to be interrupted, which in turn interrupts the WAL log reading process. At the same time, master-slave failover triggers a change in the WAL log timeline, and the timeline changes incrementally.

[0004] However, when the synchronization tool attempts to rebuild the WAL log reading process after the switchover, log reading anomalies may occur because the log timeline has changed. Current technology typically involves manually copying the missing WAL log files from the database and manually specifying new read breakpoints to rebuild the log reading chain. Therefore, the existing method of manually copying missing logs and specifying new breakpoints suffers from low efficiency and poor real-time performance. Summary of the Invention

[0005] This application provides a log data reading method, apparatus, device, medium, and program product to solve the technical problems of low efficiency and poor real-time performance in the prior art.

[0006] Firstly, this application provides a method for reading log data, the method comprising:

[0007] Obtain the mapping relationship between multiple timeline identifiers and their corresponding starting log sequence numbers in the database; the timeline identifiers are generated during database master-slave switchover.

[0008] Based on the mapping relationship, the start log sequence number and the end log sequence number of the log data, determine whether the log data spans different timeline identifiers;

[0009] When the log data spans different timeline identifiers, the log reading process is reconstructed based on the starting log sequence number of the new timeline identifier, where the new timeline identifier is different from the multiple timeline identifiers.

[0010] In one possible implementation, determining whether the log data spans different timeline identifiers based on the mapping relationship, the start log sequence number, and the end log sequence number of the log data includes:

[0011] Obtain the start log sequence number and end log sequence number of the log data;

[0012] Based on the start log sequence number and the end log sequence number, determine the log file name and timeline identifier corresponding to the log data;

[0013] If there are inconsistencies in the log file name and / or timeline identifier corresponding to the log data, it is determined that the log data spans different timeline identifiers.

[0014] In one possible implementation, before determining whether the log data crosses different timeline identifiers based on the mapping relationship, the start log sequence number, and the end log sequence number of the log data, the method further includes:

[0015] The mapping between the most recently accessed starting log sequence number and the log file name is stored through a caching mechanism;

[0016] The step of determining the log file name and timeline identifier corresponding to the log data based on the start log sequence number and the end log sequence number includes:

[0017] When determining the log file name and timeline identifier corresponding to the log data based on the start log sequence number and the end log sequence number, the log file name corresponding to the start log sequence number of the log data is first retrieved from the cache.

[0018] In one possible implementation, the log reading process reconstructing based on the starting log sequence number of the new timeline identifier includes:

[0019] Based on the starting log sequence number of the new timeline identifier, the log reading process is reinitialized through the physical replication slot interface;

[0020] Specify the new timeline identifier and the starting log sequence number in the physical replication slot interface to instruct the database to send log data from the starting position of the new timeline.

[0021] In one possible implementation, the method further includes:

[0022] The log data splitting and writing operations are handled using an asynchronous thread pool;

[0023] The task is rebuilt by asynchronously triggering the log data reading process through a message queue.

[0024] In one possible implementation, the method further includes:

[0025] The frequency and batch size of log reading in the read process reconstruction task are dynamically adjusted according to the real-time load status of the database; wherein, the batch size of a single read is increased when the load is low, and the reading frequency is reduced when the load is high.

[0026] In one possible implementation, obtaining the mapping relationship between multiple timeline identifiers and corresponding starting log sequence numbers in the database includes:

[0027] Query all timeline identifiers and their corresponding starting log sequence numbers using the database system tables;

[0028] The mapping relationship between each timeline identifier and the starting log sequence number is obtained by calling a predefined function through the database management interface.

[0029] In one possible implementation, the method further includes:

[0030] Verify the validity of the new timeline identifier;

[0031] If the new timeline identifier is valid, the reconstructed log reading process continues.

[0032] If the new timeline identifier is invalid, an alarm is triggered and the system rolls back to the most recent consistency breakpoint; the consistency breakpoint is a known complete and continuous read position in the log data.

[0033] Secondly, this application provides a log data reading device, comprising:

[0034] The acquisition module is used to acquire the mapping relationship between multiple timeline identifiers and their corresponding starting log sequence numbers in the database; the timeline identifiers are identifiers generated during database master-slave switchover.

[0035] The judgment module is used to determine whether the log data crosses different timeline identifiers based on the mapping relationship, the start log sequence number and the end log sequence number of the log data;

[0036] The reconstruction module is used to reconstruct the log reading process based on the starting log sequence number of the new timeline identifier when the log data spans different timeline identifiers, wherein the new timeline identifier is different from the multiple timeline identifiers.

[0037] In one possible implementation, the log data reading device further includes: a determination module and a judgment module;

[0038] The acquisition module is used to acquire the start log sequence number and end log sequence number of the log data;

[0039] The determining module is used to determine the log file name and timeline identifier corresponding to the log data based on the start log sequence number and the end log sequence number;

[0040] The determination module is used to determine that the log data spans different timeline identifiers if there is an inconsistency between the log file name and / or the timeline identifier corresponding to the log data.

[0041] In one possible implementation, the log data reading device further includes: a storage module;

[0042] The storage module is used to store the correspondence between the most recently accessed starting log sequence number and the log file name through a caching mechanism;

[0043] The determining module is further configured to determine the log file name and timeline identifier corresponding to the log data based on the start log sequence number and the end log sequence number, including:

[0044] When determining the log file name and timeline identifier corresponding to the log data based on the start log sequence number and the end log sequence number, the log file name corresponding to the start log sequence number of the log data is first retrieved from the cache.

[0045] In one possible implementation, the log data reading device further includes: a reading module and a sending module;

[0046] The reading module is used to reinitialize the log reading process through the physical replication slot interface according to the starting log sequence number of the new timeline identifier.

[0047] The sending module is used to specify the new timeline identifier and the starting log sequence number in the physical replication slot interface, so as to instruct the database to send log data from the starting position of the new timeline.

[0048] In one possible implementation, the log data reading device further includes: a processing module and a triggering module;

[0049] The processing module is used to process the splitting and writing operations of the log data through an asynchronous thread pool;

[0050] The triggering module is used to asynchronously trigger the log data reading process reconstruction task through a message queue.

[0051] In one possible implementation, the log data reading device further includes: an adjustment module;

[0052] The adjustment module is used to dynamically adjust the frequency and batch size of log reading in the read process reconstruction task according to the real-time load status of the database; wherein, the batch size of a single read is increased when the load is low, and the reading frequency is reduced when the load is high.

[0053] In one possible implementation, the log data reading device further includes: a query module;

[0054] The query module is used to query all timeline identifiers and their corresponding starting log sequence numbers through the database system table;

[0055] The acquisition module is used to obtain the mapping relationship between each timeline identifier and the starting log sequence number by calling a predefined function through the database management interface.

[0056] In one possible implementation, the log data reading device further includes: a verification module and an execution module;

[0057] The verification module is used to verify the legality of the new timeline identifier;

[0058] The execution module is configured to continue executing the reconstruction log reading process if the new timeline identifier is valid.

[0059] The triggering module is also used to trigger an alarm and roll back to the most recent consistency breakpoint if the new timeline identifier is invalid; the consistency breakpoint is a known complete and continuous read position in the log data.

[0060] Thirdly, embodiments of this application provide a log data reading device, including: a memory and a processor;

[0061] The memory stores computer-executed instructions;

[0062] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0063] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0064] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0065] The log data reading method provided in this application establishes a mapping relationship between timeline identifiers and starting log sequence numbers to determine the timeline span of log data. When a new timeline is generated during a database master-slave switch, the log reading process can be automatically reconstructed based on the starting log sequence number corresponding to the new timeline identifier, without manual intervention. This solves the pain points of low efficiency and poor real-time performance of traditional manual reconstruction methods, and effectively avoids the error risks that may be introduced by manual operation. It ensures the continuity of log data reading between the source and target ends during the master-slave switch, thereby guaranteeing the consistency and reliability of data synchronization between the database master and slave nodes, and improving the high availability and ease of operation and maintenance of the database system in the master-slave switch scenario. Attached Figure Description

[0066] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0067] Figure 1 Flowchart of the log data reading method provided in this application Figure 1 ;

[0068] Figure 2 Flowchart of the log data reading method provided in this application Figure 2 ;

[0069] Figure 3 Flowchart of the log data reading method provided in this application Figure 3 ;

[0070] Figure 4 A schematic diagram of the log data reading device provided in this application;

[0071] Figure 5 A schematic diagram of the log data reading device provided in this application.

[0072] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0073] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0074] It should be noted that the log data reading method, apparatus, device, medium and program products provided in this application can be used in the field of big data, or in any field other than big data. This application does not limit the application field of the log data reading method, apparatus, device, medium and program products.

[0075] In a real-time data synchronization solution based on database write-ahead logs, in order to read the write-ahead logs from the source database in real time, the log data is written to a local file, parsed and processed, and then written to the target database, thereby ensuring data consistency between the source and target.

[0076] In existing technologies, log reading relies on creating physical replication slots within the database and using the physical replication slot interface provided by the JDBC driver to achieve real-time log retrieval. This is currently the mainstream technical path for database log synchronization. In database master-slave failover scenarios, the database needs to undergo a restart process, which directly causes the connection between the synchronization tool and the database to be interrupted, thus interrupting the WAL log reading process. Simultaneously, the master-slave failover triggers a change in the WAL log timeline, and the timeline changes incrementally. When the synchronization tool attempts to rebuild the WAL log reading process after the failover, log reading anomalies occur because the log timeline has changed, making it impossible to directly restore the normal log retrieval chain. Existing technologies address this by manually copying the missing WAL log files from the database during the failover and manually specifying new read breakpoints to rebuild the log reading chain.

[0077] However, this manual operation method has obvious drawbacks such as low efficiency and poor real-time performance. It not only increases the operation cost for maintenance personnel, but also causes a long delay in data synchronization, making it difficult to meet the data synchronization requirements in high availability scenarios.

[0078] To address the aforementioned issues, the log data reading method provided in this application pre-obtains the mapping relationship between multiple timeline identifiers (exclusive identifiers generated during master-slave switchover) in the database and their corresponding starting log sequence numbers. Then, based on this mapping relationship and the starting and ending log sequence numbers of the log data to be processed, it determines whether the log data spans different timeline identifiers. If a span is determined, the log reading process is automatically reconstructed based on the starting log sequence number of the new timeline identifier, thereby replacing the traditional manual reconstruction method. This effectively avoids the shortcomings of low efficiency and poor real-time performance of manual operations, ultimately ensuring the continuity and consistency of data synchronization between the database source and target ends.

[0079] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0080] Figure 1 Flowchart of the log data reading method provided in this application Figure 1 In this embodiment, the executing entity is, for example, a log data reading system. Figure 1 As shown, the method includes:

[0081] S101: Obtain the mapping relationship between multiple timeline identifiers and their corresponding starting log sequence numbers in the database; the timeline identifiers are generated during database master-slave switchover.

[0082] The timeline identifier is a unique identifier automatically generated by the system when a primary / standby switch occurs, used to distinguish the database runtime sequence within different primary / standby switch cycles.

[0083] Master-slave failover is a core operation in a high-availability database architecture. When the master database fails or needs maintenance, the standby database will take over the services of the master database. At this time, the system will create a new timeline identifier to avoid confusion of time-series information such as logs and data before and after the failover.

[0084] The starting log sequence number is the starting number of the database log for each timeline. The log sequence number is a unique identifier for the database log, incrementing in the order of log generation, and is used to locate and trace log records.

[0085] The mapping relationship is a data structure that associates each timeline identifier with its corresponding starting log sequence number.

[0086] Specifically, first, determine the data acquisition source, which can typically be extracted from the database's system metadata table, log management module, or high availability switchover log file. The system metadata table persistently stores the timeline identifier generated for each master-slave switchover, as well as the starting log sequence number of the first log entry after the switchover is completed; the log management module records the timeline switchover events and their corresponding starting log sequence numbers in real time during the switchover process; and the high availability switchover log file retains a complete log of the switchover process, including the association information between the timeline identifier and the starting log number.

[0087] Secondly, when performing query or read operations, if the data is obtained from the metadata table, it can be extracted by executing a preset SQL query statement; if the data is obtained from the log file, the database can be used to parse and switch records and extract related data through the log reading tool provided by the database.

[0088] Finally, the extracted data is processed to remove duplicate or invalid records (such as temporary timeline identifiers that have not been switched over), and a set of key-value pair mapping relationships (such as a dictionary or hash table) is constructed, where the key is the timeline identifier and the value is the corresponding starting log sequence number.

[0089] S102: Based on the mapping relationship, the start log sequence number and the end log sequence number of the log data, determine whether the log data spans different timeline identifiers.

[0090] The end log sequence number is a unique number of the last log entry in the current log segment that needs to be processed, marking the end position of log processing.

[0091] The "Across different timeline identifiers" refers to the range of log data to be processed (from the start log sequence number to the end log sequence number). This range includes not only log records within a specific timeline but also log records within other timelines. In other words, the log range covers the log intervals corresponding to at least two timeline identifiers.

[0092] Specifically, based on the starting log sequence number corresponding to each timeline identifier in the mapping relationship, the log sequence number range of each timeline can be divided. Since the log sequence number increases in the order of generation, the ending log sequence number of the previous timeline is the starting log sequence number of the next timeline minus 1 (assuming no log loss). Therefore, the log range of each timeline is [the starting log sequence number of the current timeline, the starting log sequence number of the next timeline - 1], and the log range of the last timeline is [its starting log sequence number, infinity) (until the next switch).

[0093] Based on the starting log sequence number of the log to be processed, find the largest starting log sequence number that is less than or equal to the starting sequence number in the mapping relationship. The corresponding timeline identifier is the starting timeline. Similarly, based on the ending log sequence number of the log to be processed, find the largest starting log sequence number that is less than or equal to the ending sequence number. The corresponding timeline identifier is the ending timeline.

[0094] Compare the start timeline identifier and the end timeline identifier: If they are the same, it means that the sequence number range of the log data to be processed falls entirely within the log interval of the same timeline and does not cross different timelines; if they are different, it means that the start part of the log to be processed falls within the interval of the start timeline and the end part falls within the interval of the end timeline, that is, the log data crosses different timeline identifiers.

[0095] S103: When log data spans different timeline identifiers, the log reading process is reconstructed based on the starting log sequence number of the new timeline identifier, where the new timeline identifier is different from multiple timeline identifiers.

[0096] Among them, the new timeline identifier refers to a new timeline identifier that is added in addition to the multiple timeline identifiers that the current log data to be processed spans. It is usually generated when the database switches from primary to backup again during the log processing process. It is not the same as the multiple timeline identifiers obtained in step S101.

[0097] Reconstructing the log reading process means that when a timeline switch occurs during the log reading process (i.e., crossing different timelines and a new timeline identifier appears), the current log reading logic based on the original timeline is terminated, the log reading-related configuration, connection, and status information are reinitialized, and a new log reading process is started based on the starting log sequence number corresponding to the new timeline identifier, ensuring the continuity and accuracy of log reading.

[0098] Specifically, after determining that the log data spans different timeline identifiers, a further check is performed to see if any new timeline identifiers exist. If new timeline identifiers exist, the log reading process is rebuilt; if it only spans multiple existing timeline identifiers and no new timelines are added, no rebuilding is required, and the logs can simply be read in chronological order.

[0099] Secondly, obtain the new timeline identifier and its corresponding starting log sequence number. By executing a metadata query statement, extract the new timeline identifier and its corresponding starting log sequence number, and update them to the original mapping relationship to improve the association data between the timeline and the starting log number.

[0100] Then, the log reading process is terminated and cleaned up. The currently running log reading thread or process is stopped, resources related to the original timeline log reading (such as log file handles, network connections, cache space, etc.) are released, and reading status records (such as the position of the read log sequence number, processing progress, etc.) are cleaned up to avoid resource leaks and state chaos.

[0101] Finally, rebuild the log reading process and restart reading. Based on the starting log sequence number corresponding to the new timeline identifier, reconfigure the log reading parameters (such as setting the reading start position to the starting log sequence number of the new timeline, specifying the log storage path corresponding to the new timeline, etc.), initialize the reading tools or components (such as recreating the log reader instance, establishing a log reading connection with the new master database, etc.), update the reading progress record (set the starting progress to the starting log sequence number of the new timeline), and then start the new log reading process to continue reading log data from the starting log sequence number of the new timeline, ensuring that log reading can seamlessly connect to log content after crossing timelines.

[0102] The log data reading method provided in this embodiment obtains the mapping relationship between multiple timeline identifiers and their corresponding starting log sequence numbers in the database. The timeline identifiers are generated during database master-slave switchover. Based on the mapping relationship, the starting log sequence number and the ending log sequence number of the log data, it is determined whether the log data spans different timeline identifiers. If the log data spans different timeline identifiers, the log reading process is reconstructed based on the starting log sequence number of the new timeline identifier. The new timeline identifier is different from the multiple timeline identifiers. This method realizes the automatic reconstruction of the log reading process after master-slave switchover, avoids the problems of low efficiency and poor real-time performance caused by manual operation, and ensures the continuity and consistency of data synchronization between the source and target ends.

[0103] Figure 2 Flowchart of the log data reading method provided in this application Figure 2 ,like Figure 2 As shown, in this embodiment... Figure 1 Based on the embodiments, the log data reading method is described in detail, which includes:

[0104] S201: Query all timeline identifiers and their corresponding starting log sequence numbers through the database system tables.

[0105] Among them, the database system table is a special data table built into the database kernel. It is automatically maintained by the system and is dedicated to storing the database's metadata, configuration information, and operating status. It is the authoritative carrier for obtaining the core information of the underlying database. The timeline identifier is a unique identifier generated during database master-slave failover, used to distinguish the WAL log time sequence of different failover periods.

[0106] The Start Log Sequence Number (LSN) is a unique number for the first WAL log entry for each timeline, and it monotonically increases in the order the logs were generated.

[0107] Specifically, during the WAL log reading process startup phase, the system first connects to the database instance and calls the database query interface to execute system table query commands. For the database, it reads the timeline identifier field from the timeline history table to obtain all timeline identifiers, and simultaneously queries the starting LSN of the corresponding timeline recorded in the table. For the database, it extracts the mapping relationship between all timeline identifiers and starting log sequence numbers by querying the thread number (corresponding timeline identifier) ​​and starting change number (corresponding starting LSN) fields in the archived log view, and caches the query results in a key-value pair set in local memory, where the key is the timeline identifier and the value is the corresponding starting LSN.

[0108] S202: Obtain the mapping relationship between each timeline identifier and the starting log sequence number by calling a predefined function through the database management interface.

[0109] Among them, the database management interface is a standardized interface provided by the database for performing management operations. Common forms include SQL function interfaces, JDBC / ODBC extension interfaces, etc.

[0110] Predefined functions are built into the database to retrieve specific metadata. They do not require user definition and can be called directly.

[0111] The mapping relationship refers to the one-to-one correspondence between timeline identifiers and starting log sequence numbers.

[0112] Specifically, based on the retrieved list of timeline identifiers, a predefined database function is called via the JDBC driver. This function retrieves the starting log sequence number of a timeline, passing each retrieved timeline identifier as a parameter. The function returns the starting LSN corresponding to that timeline. For databases supporting partitioned namespace solid-state drive storage engines, a dedicated management function can be called to retrieve the mapping relationship between timelines and log sequence numbers, directly returning a set of mappings between all timeline identifiers and their starting LSNs. The function's return result is cross-validated with the query result to ensure data consistency. Finally, a complete and reliable timeline-starting LSN mapping table is generated and stored in a local persistent cache to avoid duplicate queries.

[0113] S203: Obtain the start log sequence number and end log sequence number of the log data.

[0114] The starting log sequence number is the identifier of the starting position of the current log segment.

[0115] The end log sequence number is an identifier indicating the end position of the current log segment.

[0116] Specifically, the WAL log data stream is obtained from the database in real time through physical replication slot interfaces (such as the database's logical log receiving interface or the database's log mining interface), and the received binary log data is transmitted to the local parsing module. The parsing module extracts the metadata fields from the log header according to the database's WAL log format specification, including two key fields: the start log sequence number and the end log sequence number. For compressed log data, decompression is required before parsing. The parsed start and end log sequence numbers are converted to standard decimal or hexadecimal format and bound to the current log segment.

[0117] S204: Determine the log file name and timeline identifier corresponding to the log data based on the start log sequence number and end log sequence number.

[0118] Specifically, first, the local log file name conversion function is called, taking the start log sequence number and end log sequence number as input. Based on the fixed size of the database WAL log file (e.g., the database default is 16MB), the WAL log file names corresponding to the two LSNs are calculated.

[0119] Then, query the generated timeline-start log sequence number mapping table and determine the timeline identifier to which the start log sequence number and end log sequence number belong by range matching.

[0120] Finally, record the source log file name and source timeline identifier corresponding to the starting log sequence number, and the target log file name and target timeline identifier corresponding to the ending log sequence number.

[0121] Optionally, a caching mechanism can be used to store the mapping between the most recently accessed starting log sequence number and the log file name;

[0122] Based on the start log sequence number and end log sequence number, determine the log file name and timeline identifier corresponding to the log data, including:

[0123] When determining the log file name and timeline identifier corresponding to the log data based on the start log sequence number and end log sequence number, the log file name corresponding to the start log sequence number of the log data is retrieved from the cache first.

[0124] Specifically, a remote dictionary service or local memory cache is first selected as the caching component, with a reasonable cache expiration time (e.g., 1 hour) and maximum capacity (e.g., 10,000 records). An LRU (Least Recently Used) eviction policy is also set. After obtaining the mapping between the starting log sequence number (LSN) and the log file name through a database system table query or a predefined function, the key-value pair (key: starting LSN, value: log file name) is immediately written to the cache to achieve cached storage of recently accessed data. When it is necessary to determine the log file name and timeline identifier based on the starting and ending LSNs, the starting LSN of the log to be processed is extracted to construct a cache query key, and the cache query interface is called to retrieve the corresponding value. If the cache is hit, the log file name is directly obtained; if the cache is not hit, the original database query logic is executed to obtain the log file name, and the new key-value pair is written to the cache. Then, the timeline identifier is determined by combining the cached or queried log file name with the pre-obtained mapping between the timeline identifier and the starting LSN, ensuring that caching is prioritized to improve query efficiency while guaranteeing data accuracy.

[0125] S205: If there is an inconsistency between the log file name and / or the timeline identifier corresponding to the log data, it is determined that the log data spans different timeline identifiers.

[0126] Among them, the identifier that crosses different timelines means that the start log sequence number and end log sequence number of the WAL log segment belong to two different timelines, which is usually triggered by database master-slave switchover.

[0127] Inconsistent filenames refer to different WAL log files corresponding to the start log sequence number and the end log sequence number, which is a cross-file scenario.

[0128] Inconsistent timeline identifiers refer to the start log sequence number and the end log sequence number corresponding to different timelines, which is a cross-timeline scenario.

[0129] Specifically, the source log file name is compared with the target log file name, and the source timeline identifier is also compared with the target timeline identifier. A cross-timeline scenario is determined if either of the following conditions is met: ① Source timeline identifier ≠ Target timeline identifier; ② Source log file name ≠ Target log file name, and the timeline corresponding to the target log file name is inconsistent with the source timeline, as verified by the timeline mapping table. If only the source log file name ≠ Target log file name, but the timeline identifiers are the same, it is determined as a simple cross-file scenario, and the timeline switching logic is not triggered. The determination result will be used as the basis for subsequent process branches and stored in the log processing context object.

[0130] S206: When log data spans different timeline identifiers, the log reading process is reinitialized through the physical replication slot interface based on the starting log sequence number of the new timeline identifier.

[0131] The new timeline identifier refers to the timeline identifier to which the end log sequence number belongs, and is the identifier of the new log branch generated after the primary / standby switch.

[0132] Reinitializing the log reading process means closing the current WAL log stream replication connection based on the original timeline, re-establishing the connection based on the new timeline, and configuring the reading parameters to ensure the continuity of log reading.

[0133] The physical replication slot interface is an interface provided by the database for establishing physical stream replication.

[0134] Specifically, when a cross-timeline scenario is identified, the new timeline identifier (i.e., the target timeline identifier) ​​is first extracted from the log processing context. Then, the generated timeline-starting LSN mapping table is queried to obtain the starting LSN corresponding to the new timeline. Next, the close method of the physical replication slot interface is called to terminate the current WAL log stream transmission connection and release related resources (such as network connections and memory buffers). Afterward, the initialization method of the physical replication slot interface is called, passing in the new timeline identifier and the corresponding starting LSN as core parameters, and configuring parameters such as the log transmission format (e.g., binary format) and compression method to complete the initialization of the new log reading process.

[0135] S207: Specify the new timeline identifier and starting log sequence number in the physical replication slot interface to instruct the database to send log data from the starting position of the new timeline.

[0136] Among them, the physical replication slot interface parameters specify the core parameters for passing log reading to the database through the interface, ensuring that the database accurately identifies the starting position of log sending.

[0137] The starting position of the new timeline refers to the LSN position of the first WAL log data corresponding to the new timeline, which is the starting point of the new log branch after the master-slave switch.

[0138] Specifically, after initializing a new physical replication slot connection, the new timeline identifier and the new timeline starting LSN are written into the connection parameter set through the interface parameter setting method. For relational databases, the timeline and starting log sequence number parameters can be specified through the start replication command, in the format: start replication slot: replication slot name; timeline: new timeline identifier, starting log sequence number: starting log sequence number value; for databases that support partition namespace solid-state drives, additional storage engine optimization parameters (such as partition namespace block alignment parameters) can be specified.

[0139] After the parameters are set, a connection confirmation command is sent to the database. The database will then start streaming write-ahead log data from the beginning of the new timeline according to the specified timeline and the starting log sequence number, ensuring that the logs read subsequently are completely consistent with the new log branch after the master-slave switchover.

[0140] S208: Handle log data splitting and writing operations through an asynchronous thread pool.

[0141] The asynchronous thread pool is a component that manages multiple asynchronous threads to process independent tasks in parallel and improve log processing efficiency.

[0142] Log data splitting refers to dividing WAL log segments that span files or timelines into multiple valid log sub-segments according to WAL file boundaries or timeline boundaries.

[0143] The write operation refers to writing the split log fragments to the local WAL log file to complete the persistent storage of the logs.

[0144] Specifically, after determining that the scenario involves cross-file or cross-timeline operations, the log splitting and writing task is encapsulated into an asynchronous task object. The task parameters include the original log fragment, start log sequence number, end log sequence number, source / target filename, and source / target timeline identifier. The task object is submitted to a pre-defined asynchronous thread pool. The thread pool automatically allocates idle threads to execute the task based on the core thread count and maximum thread count configuration. During thread execution, the log fragments that cross files are first split into a first half (belonging to the source file) and a second half (belonging to the target file) according to the size boundaries of the WAL file. If it is a cross-timeline scenario, the split point is the maximum LSN position of the original timeline. After splitting, the two sub-fragments are written to their respective local WAL files, and the local log read breakpoint is updated (recording the maximum LSN already written). After the task execution is complete, a callback function is used to report the result to the main process, ensuring that the main process is aware of the log writing status.

[0145] S209: Rebuild the task by asynchronously triggering the log data reading process through the message queue.

[0146] Among them, the read process reconstruction task refers to the task of re-establishing the WAL log read process when a timeline switch occurs, which is a key task to ensure continuous log reading.

[0147] Specifically, when a cross-timeline scenario is identified, the main process generates a process reconstruction task message. The message content includes core information such as the new timeline identifier, the new timeline's starting LSN, and the current database connection parameters. This message is sent to a pre-defined message queue (such as Kafka or RabbitMQ), which persistently stores the task message to prevent task loss. Simultaneously, a message consumer thread is started to continuously listen for process reconstruction task messages in the message queue. When the consumer thread receives the message, it parses the message content and calls the methods in steps S206 and S207 to complete the log reading process reconstruction under the new timeline. Asynchronously triggering the task through the message queue avoids blocking the main process and implements a task retry mechanism. If process reconstruction fails, the message queue can re-deliver the task, improving system reliability.

[0148] S210: Dynamically adjust the frequency and batch size of log reading in the read process reconstruction task according to the real-time load status of the database; that is, increase the batch size of a single read when the load is low, and reduce the reading frequency when the load is high.

[0149] Real-time load status refers to the current resource usage of the database, including indicators such as CPU utilization, memory usage, disk I / O load, and transaction volume.

[0150] Log read frequency refers to the number of times WAL logs are retrieved from the database per unit of time.

[0151] Batch size refers to the size of a WAL log segment read from the database in a single transaction, usually measured in LSN range length or number of bytes.

[0152] Specifically, in the log reading process, a new load monitoring thread is added. This thread periodically queries the database's real-time load metrics through a database management interface (such as JDBC database metadata), with a query cycle configurable to 5 seconds. Load states are divided into three levels: low load (CPU utilization <30%, disk I / O utilization <20%), medium load (CPU utilization 30%-70%, disk I / O utilization 20%-60%), and high load (CPU utilization >70%, disk I / O utilization >60%). When a low load state is detected, the batch read parameters of the physical replication slot interface are adjusted, increasing the LSN interval length for a single read from the default 1000 units to 5000 units to reduce read frequency and network interaction overhead. When a high load state is detected, the LSN interval length for a single read is reduced to 500 units, and the read interval is extended (from 1 second to 3 seconds) to reduce database resource consumption. The default parameters are maintained under medium load conditions. The load adjustment strategy is dynamically loaded through a configuration file, supporting customized parameter thresholds based on different database types and hardware environments.

[0153] The log data reading method provided in this embodiment queries the full timeline identifier and its corresponding starting log sequence number through the database system table, then uses the database management interface to call a predefined function to construct the mapping relationship between the two, and then obtains the starting log sequence number and ending log sequence number of the log data to be processed. Based on this, the log file name and timeline identifier corresponding to the log data are determined. If there is an inconsistency in the log file name and / or timeline identifier, it is determined that the log data spans different timeline identifiers. In the case that the log data spans timelines, the log reading stream is reinitialized through the physical replication slot interface based on the starting log sequence number of the new timeline identifier. The process involves specifying a new timeline identifier and the corresponding starting log sequence number in the interface to instruct the database to send log data from the starting position of the new timeline. An asynchronous thread pool is used to handle the splitting and writing of log data. A message queue is used to asynchronously trigger the log reading process reconstruction task, and the frequency and batch size of log reading in the reconstruction task are dynamically adjusted according to the real-time load of the database. The batch size of a single read is increased when the load is low, and the reading frequency is reduced when the load is high. Ultimately, this achieves automatic reconstruction of the log reading process after a master-slave switch, avoiding the low efficiency and poor real-time performance issues of manual operation, and ensuring the continuity and consistency of data synchronization between the source and target ends.

[0154] Figure 3 Flowchart of the log data reading method provided in this application Figure 3 ,like Figure 3 As shown, in this embodiment... Figure 2 Based on the embodiments, the validity verification of the new timeline identifier is described in detail. The method includes:

[0155] S301: Verify the validity of the new timeline identifier.

[0156] Specifically, format validation needs to verify whether the new timeline identifier conforms to the format specified by the database (e.g., timeline identifiers in the database are non-negative integers); validity validation needs to confirm that the identifier exists in the retrieved system timeline records and that the corresponding starting log sequence number (LSN) is valid. Secondly, a predefined validation function is called through the database management interface, or the system timeline table is queried directly to compare the format and existence of the new timeline identifier; simultaneously, it is verified whether the starting LSN corresponding to the identifier is greater than 0 (or within the valid range of database LSNs). Finally, the validation results are recorded, storing information about successful or failed validations in the log.

[0157] S302: If the new timeline identifier is valid, continue the log reconstruction reading process.

[0158] Specifically, first, the core parameters corresponding to the valid new timeline identifier are obtained, i.e., the starting LSN corresponding to the identifier is extracted from the system timeline records, ensuring parameter integrity. Second, through the physical replication slot interface, the associated timeline of the replication slot is updated to the valid new timeline identifier, and the corresponding starting LSN is specified as the reading starting point; the log reader configuration (such as buffer size and timeout) is reinitialized, the old read connection is closed and resources are released, and a new log read connection is established. Finally, a new log read process is started, and the reconstruction completion status is recorded synchronously to ensure that subsequent log data is read normally from the valid position of the new timeline.

[0159] S303: If the new timeline identifier is invalid, an alarm is triggered and the system rolls back to the most recent consistency breakpoint; the consistency breakpoint is a known complete and continuous read position in the log data.

[0160] Specifically, first, an alarm message is sent via a message queue or monitoring interface. The message includes the specific value of the invalid timeline identifier, the reason for the verification failure (e.g., incorrect format, non-existent), and the current operation time. Simultaneously, detailed exception information is recorded in the local log. Second, the most recent read position marked as complete and continuous is retrieved from the local log processing records. This position must meet the following conditions: all corresponding log data has been written locally, there are no missing segments, and verification has passed (e.g., passing LSN continuity verification). Finally, the current position of the log reader is reset to the LSN corresponding to this consistency breakpoint. The read process based on the timeline to which this breakpoint belongs is reinitialized, and operations related to invalid new timeline identifiers are abandoned, ensuring that log reading returns to a stable and reliable state.

[0161] Figure 4 A schematic diagram of the log data reading device provided in this application is shown below. Figure 4 As shown, the log data reading device 400 provided in this embodiment includes:

[0162] The acquisition module 401 is used to obtain the mapping relationship between multiple timeline identifiers and their corresponding starting log sequence numbers in the database; the timeline identifiers are identifiers generated during database master-slave switchover.

[0163] The judgment module 402 is used to determine whether the log data crosses different timeline identifiers based on the mapping relationship, the start log sequence number and the end log sequence number of the log data;

[0164] Reconstruction module 403 is used to reconstruct the log reading process based on the starting log sequence number of the new timeline identifier when log data spans different timeline identifiers. The new timeline identifier is different from multiple timeline identifiers.

[0165] In one possible implementation, the log data reading device further includes: a determination module 404 and a judgment module 405;

[0166] Module 401 is used to obtain the start log sequence number and end log sequence number of the log data;

[0167] The determination module 404 is used to determine the log file name and timeline identifier corresponding to the log data based on the start log sequence number and the end log sequence number;

[0168] The determination module 405 is used to determine if the log data spans different timeline identifiers when there is an inconsistency between the log file name and / or the timeline identifier corresponding to the log data.

[0169] In one possible implementation, the log data reading device further includes: a storage module 406;

[0170] Storage module 406 is used to store the mapping between the most recently accessed starting log sequence number and the log file name through a caching mechanism;

[0171] Module 404 is further configured to determine the log file name and timeline identifier corresponding to the log data based on the start log sequence number and the end log sequence number, including:

[0172] When determining the log file name and timeline identifier corresponding to the log data based on the start log sequence number and end log sequence number, the log file name corresponding to the start log sequence number of the log data is retrieved from the cache first.

[0173] In one possible implementation, the log data reading device further includes: a reading module 407 and a sending module 408;

[0174] The read module 407 is used to reinitialize the log read process through the physical replication slot interface based on the starting log sequence number of the new timeline identifier.

[0175] The sending module 408 is used to specify a new timeline identifier and a starting log sequence number in the physical replication slot interface to instruct the database to send log data from the starting position of the new timeline.

[0176] In one possible implementation, the log data reading device further includes: a processing module 409 and a triggering module 410;

[0177] Processing module 409 is used to handle the splitting and writing operations of log data through an asynchronous thread pool;

[0178] Trigger module 410 is used to asynchronously trigger the log data reading process reconstruction task through a message queue.

[0179] In one possible implementation, the log data reading device further includes: an adjustment module 411;

[0180] The adjustment module 411 is used to dynamically adjust the frequency and batch size of log reading in the read process reconstruction task according to the real-time load status of the database; that is, the batch size of a single read is increased when the load is low, and the reading frequency is reduced when the load is high.

[0181] In one possible implementation, the log data reading device further includes: a query module 412;

[0182] Query module 412 is used to query all timeline identifiers and their corresponding starting log sequence numbers through the database system tables;

[0183] The acquisition module 401 is used to obtain the mapping relationship between each timeline identifier and the starting log sequence number by calling a predefined function through the database management interface.

[0184] In one possible implementation, the log data reading device further includes: a verification module 413 and an execution module 414;

[0185] Verification module 413 is used to verify the validity of the new timeline identifier;

[0186] Execution module 414 is used to continue the reconstruction log reading process if the new timeline identifier is valid;

[0187] The triggering module 410 is also used to trigger an alarm and roll back to the most recent consistency breakpoint if the new timeline identifier is invalid; the consistency breakpoint is a known complete and continuous read position in the log data.

[0188] The log data reading device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0189] Figure 5 A schematic diagram of the log data reading device provided in this application. Figure 5 As shown, the electronic device of this embodiment may include: at least one processor 501; and a memory 502 communicatively connected to the at least one processor; wherein the memory 502 stores instructions that can be executed by the at least one processor 501, and the instructions are executed by the at least one processor 501 to cause the electronic device to perform the method as described in any of the above embodiments.

[0190] Optionally, the memory 502 can be either standalone or integrated with the processor 501. When the memory 502 is set up independently, the device also includes a bus for connecting the memory 502 and the processor 501.

[0191] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the foregoing embodiments, and will not be repeated here.

[0192] This application also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the methods provided in any of the foregoing embodiments can be implemented.

[0193] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in any of the foregoing embodiments.

[0194] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0195] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0196] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0197] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or in the form of software program modules.

[0198] Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc.

[0199] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0200] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0201] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0202] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A log data reading method characterized by comprising: include: Obtain the mapping relationship between multiple timeline identifiers and their corresponding starting log sequence numbers in the database; The timeline identifier is an identifier generated during database master-slave failover. Based on the mapping relationship, the start log sequence number and the end log sequence number of the log data, determine whether the log data spans different timeline identifiers; When the log data spans different timeline identifiers, the log reading process is reconstructed based on the starting log sequence number of the new timeline identifier, where the new timeline identifier is different from the multiple timeline identifiers.

2. The method of claim 1, wherein, The step of determining whether the log data spans different timeline identifiers based on the mapping relationship, the start log sequence number, and the end log sequence number of the log data includes: Obtain the start log sequence number and end log sequence number of the log data; Based on the start log sequence number and the end log sequence number, determine the log file name and timeline identifier corresponding to the log data; If there are inconsistencies in the log file name and / or timeline identifier corresponding to the log data, it is determined that the log data spans different timeline identifiers.

3. The method of claim 2, wherein, Before determining whether the log data crosses different timeline identifiers based on the mapping relationship, the start log sequence number, and the end log sequence number of the log data, the method further includes: The mapping between the most recently accessed starting log sequence number and the log file name is stored through a caching mechanism; The step of determining the log file name and timeline identifier corresponding to the log data based on the start log sequence number and the end log sequence number includes: When determining the log file name and timeline identifier corresponding to the log data based on the start log sequence number and the end log sequence number, the log file name corresponding to the start log sequence number of the log data is first retrieved from the cache.

4. The method according to any one of claims 1-3, characterized in that, The log reading process, which reconstructs the log reading based on the starting log sequence number of the new timeline identifier, includes: Based on the starting log sequence number of the new timeline identifier, the log reading process is reinitialized through the physical replication slot interface; Specify the new timeline identifier and the starting log sequence number in the physical replication slot interface to instruct the database to send log data from the starting position of the new timeline.

5. The method according to claim 4, characterized in that, The method further includes: The log data splitting and writing operations are handled using an asynchronous thread pool; The task is rebuilt by asynchronously triggering the log data reading process through a message queue.

6. The method according to claim 5, characterized in that, The method further includes: The frequency and batch size of log reading in the read process reconstruction task are dynamically adjusted according to the real-time load status of the database; wherein, the batch size of a single read is increased when the load is low, and the reading frequency is reduced when the load is high.

7. The method according to any one of claims 1-3, characterized in that, The step of obtaining the mapping relationship between multiple timeline identifiers and corresponding starting log sequence numbers in the database includes: Query all timeline identifiers and their corresponding starting log sequence numbers using the database system tables; The mapping relationship between each timeline identifier and the starting log sequence number is obtained by calling a predefined function through the database management interface.

8. The method according to any one of claims 1-3, characterized in that, The method further includes: Verify the validity of the new timeline identifier; If the new timeline identifier is valid, the reconstructed log reading process continues. If the new timeline identifier is invalid, an alarm is triggered and the system rolls back to the most recent consistency breakpoint; the consistency breakpoint is a known complete and continuous read position in the log data.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8.

Images