Redo log processing method and device, storage medium and program product

Abstract

The embodiment of the invention provides a redo log processing method and device, a storage medium and a program product, and relates to the field of distribution. The method comprises the steps that metadata information of a redo log file in an ASM storage layer is obtained through an ASM API interface of an automatic storage management system; wherein the ASM API interface is used for accessing an ASM storage layer, the redo log file comprises a log file used for recording database change operation, and the metadata information at least comprises a file path, a block number, a block size and a file type; according to the metadata information, performing block reading on the redo log file through an ASM API interface to obtain redo log data read in a block manner; and converting the redo log data read in blocks into a local redo log format to obtain a local redo log. According to the method, the capability of reading and analyzing the redo log file from the ASM storage layer is increased, and the efficiency of fault recovery in database operation and maintenance is improved.

Description

Technical Field

[0001] This application relates to the field of distributed systems, and more particularly to a redo log processing method, device, storage medium, and program product. Background Technology

[0002] In the field of database operations and maintenance, redo logs are core to ensuring data security and auditability. Taking data recovery scenarios as an example, when data is accidentally deleted or a system failure occurs, the complete operation records in the redo logs are essential to accurately restore the database to a consistent state before the failure.

[0003] In current implementations, this is primarily achieved through the database's built-in log analysis tools. By converting redo log files into readable SQL statements, administrators can reverse-engineer historical data changes to pinpoint specific erroneous operations or transaction points, providing accurate information for subsequent recovery operations.

[0004] However, the above implementation method is difficult to meet the need for real-time analysis of the latest logs. Summary of the Invention

[0005] This application provides a redo log processing method, device, storage medium, and program product to solve the real-time technical problem of log analysis.

[0006] In a first aspect, embodiments of this application provide a redo log processing method, the method comprising:

[0007] Metadata information of redo log files in the ASM storage layer can be obtained through the Automatic Storage Management System (ASM) API interface. The ASM API interface is used to access the ASM storage layer. Redo log files include log files used to record database change operations. Metadata information includes at least the file path, number of blocks, block size, and file type.

[0008] Based on the metadata information, the redo log file is read in blocks through the ASM API interface to obtain the redo log data read in blocks;

[0009] The redo log data read in blocks is converted into local redo log format to obtain local redo logs.

[0010] In this embodiment, the above steps implement a redo log acquisition and transformation mechanism independent of the database instance and standby node. The entire process communicates directly with the storage layer via the ASM API, without the need for database instance involvement, achieving lightweight, low-latency log download. Simultaneously, integrated format conversion ensures that the output logs are ready to use immediately, effectively supporting downstream application scenarios such as data recovery, security auditing, and performance analysis.

[0011] In one possible implementation, metadata information of the redo log files in the ASM storage layer is obtained through the ASM API interface, including:

[0012] Query the redo thread number in the ASM storage layer;

[0013] The corresponding starting system change number (SCN) is obtained based on the redo thread number; the starting system change number (SCN) is used to determine the starting position for reading the log file.

[0014] Access the ASM storage layer via the ASM API interface and obtain the metadata information corresponding to the redo thread number based on the initial system change number (SCN).

[0015] In this implementation, the ASM API interface directly queries and retrieves metadata information from the ASM storage layer, overcoming the architectural limitations of traditional solutions that require reliance on database instances or standby nodes to access log metadata. Calling the file attribute query interface provided by the ASM API directly reads file attributes, decoupling from the database instance layer and reducing system architecture complexity and daily maintenance costs. Simultaneously, the dynamic positioning mechanism based on SCN numbers ensures that the starting position of log reading remains synchronized with the actual change state of the database, thereby improving the consistency and real-time performance of the retrieved log data.

[0016] In one possible implementation, the redo log file is read in blocks based on metadata information via the ASM API interface to obtain the block-read redo log data, including:

[0017] Assign a separate download thread to each redo thread number;

[0018] Based on an independent download thread, the redo log file is read in blocks through the ASM API interface to obtain the redo log data read in blocks. The number of blocks read each time does not exceed the preset number of allocation units (AU).

[0019] This implementation effectively solves the problem of low concurrent download efficiency in multi-instance cluster environments such as RAC by allocating an independent download thread to each redo thread. The independent thread design reduces the coupling between threads, improves fault isolation, and ensures that an anomaly in a single thread will not affect the normal operation of other threads, thereby enhancing the system stability and real-time performance of the entire log download process. Simultaneously, the block-based reading mechanism, which limits the number of allocation units read per read, avoids memory and I / O performance bottlenecks caused by excessively large amounts of data read in a single operation, further improving log download efficiency and overall response speed.

[0020] In one possible implementation, the process of obtaining the chunked redo log data by reading the redo log file through the ASM API interface using an independent download thread also includes:

[0021] Predict the next data blocks to be read based on the currently read AU sequence number;

[0022] Based on the data blocks that need to be read later, initiate asynchronous read requests in advance.

[0023] This implementation reduces I / O wait time by prefetching subsequent data blocks, avoiding performance bottlenecks caused by reading blocks one by one. This mechanism is particularly effective when reading large files, as it ensures that subsequent read requests do not need to wait for disk I / O by preloading subsequent data blocks into the memory cache, thereby improving overall download efficiency. Furthermore, the combination of asynchronous reading and caching reduces access latency at the storage layer, enabling the system to respond to log download requests more quickly.

[0024] In one possible implementation, the local redo log format is used to indicate the following contents in the redo log file header: a fixed header identifier, logical block size, number of blocks, and a magic value; the magic value is a fixed value used to identify the file format. Converting the redo log data read in blocks into the local redo log format yields the local redo log, which includes:

[0025] The redo log data read in blocks is reassembled based on the local redo format to obtain the local redo log.

[0026] This implementation resolves the compatibility issues between the ASM storage format and standard analysis tools by constructing a standard redo log file header and reassembling data blocks. The standardized format conversion mechanism ensures the usability of the output files, allowing log files to be directly used by standard analysis tools without additional processing. Furthermore, the design of the header construction and data block reassembly logic reduces the complexity of log conversion, minimizes the need for manual intervention, and improves the accuracy of automated processing.

[0027] In one possible implementation, the redo log data read in chunks is converted into a local redo log format. After obtaining the local redo log, the following steps are also included:

[0028] Detect state changes of redo log files in the ASM storage layer;

[0029] When it is detected that redo log files in the ASM storage layer are being converted to archive logs, the system automatically switches to archive log download mode.

[0030] This implementation ensures the real-time nature of online logs and the integrity of archived logs by dynamically monitoring changes in log file status and switching download modes accordingly. For example, when a log file is converted to archived status, the system can seamlessly switch to archived log download mode, avoiding data loss due to untimely log switching. Furthermore, monitoring log status changes further improves the accuracy of log switch detection, preventing anomalies caused by misjudgments of status.

[0031] In one possible implementation, the redo log data read in chunks is converted into a local redo log format. After obtaining the local redo log, the following steps are also included:

[0032] Frequently accessed metadata in the ASM storage layer is temporarily stored in a cache;

[0033] The cache invalidation policy manages the source data in the cache; the cache invalidation policy includes: automatically refreshing the contents of the cache when the SCN number of the log file changes.

[0034] In this implementation, frequently accessed metadata (such as log file path, number of blocks, and block size) is temporarily stored in a cache through a caching mechanism, and a cache invalidation policy is set (e.g., based on SCN number changes or time intervals). For example, when the SCN number of a log file is updated, the cache content is automatically refreshed to ensure the timeliness of the metadata. This mechanism can reduce repeated query requests to the ASM storage layer and reduce storage access overhead.

[0035] In a second aspect, this application provides an electronic device, including: a processor and a memory communicatively connected to the processor;

[0036] The memory stores instructions that the computer executes;

[0037] The processor executes computer-executable instructions stored in memory to implement any of the methods of the first aspect.

[0038] Thirdly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of any one of the first aspects.

[0039] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the first aspects. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0041] Figure 1A flowchart illustrating a redo log processing method provided in an embodiment of this application;

[0042] Figure 2 This is a schematic diagram illustrating the multi-threaded log reading and switching process provided in an embodiment of this application.

[0043] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0044] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0045] It should be noted that the redo log processing method, device, storage medium, and program product provided in this application can be used in the distributed field, or in any field other than distributed fields. This application does not limit the application field of the redo log processing method, device, storage medium, and program product.

[0046] Specific application scenarios for this application include various operational scenarios involving efficient and accurate offline acquisition and processing of database redo logs in complex environments where real application clusters (RAC) and automatic storage management (ASM) storage architectures coexist. Through an independent processing channel decoupled from the production database, direct access to and unified format conversion of the underlying log data are achieved, thus enabling applications in the following specific business scenarios:

[0047] Large-scale data recovery and drills: When a production database encounters logical errors, human error, or requires disaster recovery drills, the solution in this application supports the rapid and complete extraction of redo logs from all relevant instances within a specific time interval from the ASM storage layer. These log files, converted to a local format, can be directly used to build a complete data recovery chain in an independent recovery environment, perform data recovery verification accurate to the second, or execute actual emergency data repair operations.

[0048] Centralized Security Audit and Compliance Check: To meet data security regulations and internal audit requirements, the solution in this application can automatically collect all redo logs from the production database cluster. These logs are then transmitted in their entirety to a dedicated audit analysis platform. On this platform, auditors can trace historical operations, analyze behavioral patterns, and receive alerts for abnormal operations, achieving efficient and compliant security auditing.

[0049] Cross-node performance diagnostics and in-depth tuning: In complex environments with multiple instances sharing storage, performance bottlenecks often involve collaboration and resource contention among multiple instances. This application's solution provides a global transaction view for performance analysis by synchronously capturing and correlating redo logs from all database instances. Using these transformed local logs, performance issues in distributed environments, such as cross-instance transaction latency and low global cache consistency efficiency, can be identified, providing data for system tuning.

[0050] To meet the needs of the above application scenarios, existing technologies mainly obtain and provide database redo log files through the following two paths:

[0051] 1. Data Guard Node Synchronization Solution

[0052] This solution indirectly obtains logs by leveraging the log transport mechanism of the Data Guard high-availability architecture. It works by configuring and maintaining a physical or logical standby node within the database environment. Redo logs generated by the primary database or instances in a RAC environment are continuously sent to and applied to the standby node via Data Guard's log transport service. When log analysis is required, the operation is redirected to the standby node's operating system file system, accessing archived log files or online log copies synchronized to that file system as source data for offline analysis.

[0053] 2. Recovery Manager (RMAN) Export Solution

[0054] This approach utilizes the standard backup functionality of the Database Recovery Manager tool to extract log files. The implementation is as follows: the administrator connects to the target database instance via an RMAN client and executes a backup command for archived logs or online redo logs, such as using the `BACKUP ARCHIVELOG` or `BACKUP AS COPY` command. RMAN then instructs the database instance to read the target log files from its managed storage (including ASM) and output binary copies of these files to a configured backup destination, such as a local file system or network shared storage. Users can then retrieve copies of the log files from this backup destination for further processing.

[0055] However, the solution described above has the following technical problems:

[0056] Architecture complexity and high operation and maintenance costs: Existing solutions rely on a complete Data Guard backup database or database instance, resulting in redundant system components, complex deployment and maintenance, and a significant increase in operation and maintenance costs.

[0057] Insufficient real-time performance: Existing solutions cannot achieve real-time and incremental acquisition of online redo logs. Log data acquisition is delayed, making it difficult to meet the needs for immediate auditing or analysis of the latest data changes.

[0058] Restricted access to the storage layer: Existing solutions require indirect access to log files through the database instance, and cannot bypass the instance to directly read and write to the ASM storage layer, which limits the flexibility and efficiency of access and puts a load on the production system.

[0059] Non-standardized log format: Log files obtained from ASM storage retain proprietary physical formats, which are incompatible with operating system standard files and mainstream log analysis tools. They require additional complex conversions before they can be used, and lack an end-to-end standardized output process.

[0060] The redo log processing method provided in this application aims to solve the aforementioned technical problems of existing technologies. Firstly, by directly calling the standard interface (ASM API) provided by the ASM storage management layer, it enables direct access and reading of the underlying redo log files, thereby eliminating reliance on a running database instance and reducing system coupling while avoiding interference with production performance.

[0061] Secondly, for cluster environments with multiple instances, a concurrent processing mechanism supporting multi-threaded collaboration was designed, which can synchronously capture and coordinate log streams from different instances, improving the overall throughput and timeliness of log collection.

[0062] Finally, to address the compatibility issues between the raw log format and general analysis tools, a format standardization process was integrated while acquiring data. This process converts ASM's unique storage structure into redo log files that can be directly recognized by standard parsing tools, achieving a seamless transition from raw data acquisition to analysis-ready status.

[0063] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0064] First, combine Figure 1 The redo log processing method provided in the embodiments of this application will be described. Figure 1 This is a flowchart illustrating a redo log processing method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps:

[0065] S101. Obtain metadata information of redo log files in the ASM storage layer through the Automatic Storage Management System (ASM) API interface.

[0066] In this step, the system directly accesses the ASM storage layer by calling the standard management interface ASM API provided by ASM to obtain metadata information for the specified redo log file. The ASM storage layer is a physical storage structure managed by the Oracle Automatic Storage Management System, which stores various database files, including database files, control files, and redo log files. Redo log files include online redo logs and archived redo logs, used to record all data change operations in the database. Metadata information includes at least the complete file path of the redo log file in ASM, file size, number of logical blocks, block size, and file type. This can be achieved by calling ASM API functions such as `dbms_diskgroup.getfileattr`.

[0067] S102. Based on the metadata information, the redo log file is read in blocks through the ASM API interface to obtain the redo log data read in blocks.

[0068] After acquiring the metadata information, the system performs a block-based read operation on the target redo log file through the ASM API interface, based on information such as file block size and number of blocks. The read process is performed using ASM allocation units (AUs) as the basic unit, and each read operation can process one or more AUs. Specifically, the target file is first opened using the dbms_diskgroup.open function, and then the dbms_diskgroup.read function is called repeatedly to read the file content sequentially according to the logical block order until the entire file is read.

[0069] This step interacts directly with the ASM storage layer, bypassing the database instance layer, thus avoiding the occupation and dependence on running database resources. The retrieved data is raw, binary redo log data organized in the ASM storage format.

[0070] S103. Convert the redo log data read in blocks into the local redo log format to obtain the local redo log.

[0071] This step restructures and converts the raw binary data obtained in S102 into a redo log file format that conforms to Oracle's standard specifications, namely the local redo log format.

[0072] The conversion process mainly includes: constructing a fixed file header according to the standard redo log format specification, which contains specific identification information, logical block size, number of blocks, and other metadata; then, reorganizing and splicing the binary data read in blocks according to the standard logical block size, removing ASM-specific storage structure information, and forming a continuous and standardized redo log data body; finally, writing the reconstructed file header and data body into the local operating system's file system to generate a standard redo log file that can be directly recognized and used by various log analysis tools.

[0073] The redo log processing method provided in this embodiment, through the aforementioned steps S101 to S103, implements a redo log acquisition and conversion mechanism independent of the database instance and standby node. The entire process communicates directly with the storage layer via the ASM API, without the involvement of the database instance, achieving lightweight and low-latency log download. Simultaneously, through integrated format conversion, it ensures that the output logs are ready to use immediately, effectively supporting downstream application scenarios such as data recovery, security auditing, and performance analysis.

[0074] Optionally, this application provides a specific implementation of a redo log processing method. This method introduces dynamic positioning of the redo thread and the starting change point during the metadata acquisition stage to improve the accuracy and automation of log acquisition.

[0075] Specifically, when retrieving metadata information from redo log files via the ASM API interface, the following steps are performed:

[0076] Step A: Query the redo thread number in the ASM storage layer.

[0077] In this step, the redo thread ID for the current environment is identified by directly accessing the database configuration and log structure information stored in the ASM storage layer. The redo thread ID is a unique identifier in the database used to distinguish logs generated by different instances. For example, in a RAC cluster, each active instance corresponds to a unique thread ID. This information can be obtained from the instance-thread mapping maintained internally by the ASM storage layer, or determined by reading metadata exported from the persistent global database view in the storage layer. The entire process does not depend on a running database instance.

[0078] Step B: Obtain the corresponding starting system change number (SCN) based on the redo thread number.

[0079] Based on the obtained redo thread number, the starting system change number associated with that thread is further determined. The system change number is a sequential number used internally by the database to mark the logical order of data changes, and the starting SCN defines the logical starting position for reading redo logs from that thread. This starting SCN can be obtained by parsing the correspondence between the log sequence recorded in the ASM storage layer and the SCN, or it can be dynamically calculated based on the recovery time point or analysis start range specified by the user.

[0080] Step C: Access the ASM storage layer through the ASM API interface and obtain the metadata information corresponding to the redo thread number based on the initial system change number.

[0081] After identifying the redo thread ID and starting SCN, the system interacts directly with the ASM storage layer via the ASM API to locate and retrieve metadata information for redo log files that meet the criteria. Specifically, this is achieved by calling ASM functions such as `dbms_diskgroup.getfileattr`, passing in a file identifier determined by the thread ID, SCN range, and log type, to directly obtain the file's detailed storage attributes, including the file's full path within the ASM disk group, the number of logical blocks, block size, and file type. This operation is completed directly at the storage layer, bypassing the database instance layer, thus achieving real-time metadata retrieval and architecture independence.

[0082] By directly querying and retrieving metadata information from the ASM storage layer via the ASM API interface, this overcomes the architectural limitations of traditional solutions that require reliance on database instances or standby nodes to access log metadata. Calling the file attribute query interface provided by the ASM API to directly read file attributes decouples the system from the database instance layer, reducing system architecture complexity and daily maintenance costs. Simultaneously, the dynamic positioning mechanism based on SCN numbers ensures that the starting position of log reading remains synchronized with the actual change state of the database, thereby improving the consistency and real-time performance of the acquired log data.

[0083] Optionally, this application embodiment also provides another specific implementation of the redo log processing method, which adopts a concurrent execution mechanism in the block reading stage to improve log download efficiency and system resource utilization in a multi-instance environment.

[0084] Specifically, during the process of reading redo log files in blocks based on metadata information via the ASM API interface, the following steps are performed:

[0085] Step A: Assign a separate download thread to each redo thread number.

[0086] In this step, the system creates and assigns an independent download thread for each redo thread number. Each download thread, as an independent execution unit, is specifically responsible for handling the reading tasks of all log files from its corresponding redo thread. For example, in a RAC cluster environment, if thread numbers 1, 2, and 3 correspond to three different instances, the system will create three download threads accordingly to process the redo logs from these three instances in parallel. This mechanism ensures that logs generated by different instances can be processed concurrently in a multi-instance environment, thereby effectively utilizing multi-core CPU and I / O resources and avoiding the performance bottleneck caused by single-threaded serial processing.

[0087] Step B: Based on an independent download thread, the redo log file is read in blocks through the ASM API interface, wherein the number of blocks read each time does not exceed the preset number of allocation units.

[0088] Within each download thread, block read operations are performed via the ASM API interface based on the metadata information (such as file path, block size, and number of blocks) of the corresponding redo log file. Read operations are performed in ASM allocation units. The system presets a maximum number of AUs that can be processed in a single read operation, for example, no more than 63 AUs, to avoid memory pressure or I / O latency caused by excessive data volume read in a single operation.

[0089] Specifically, after the download thread calls `dbms_diskgroup.open` to open the target file, it repeatedly calls the `dbms_diskgroup.read` function, reading one or more Active Directory (AU) at a time, until all data blocks of the redo log file are completely read. This process is executed independently and concurrently in each download thread, and the read operations between different threads do not interfere with each other, thus forming a highly efficient parallel download architecture.

[0090] By allocating a separate download thread to each redo thread, the low concurrent download efficiency issue in multi-instance cluster environments such as RAC is effectively resolved. The independent thread design reduces coupling between threads, improves fault isolation, and ensures that an anomaly in a single thread will not affect the normal operation of other threads, thereby enhancing the system stability and real-time performance of the entire log download process. Simultaneously, the block-based reading mechanism, which limits the number of allocated units read per operation, avoids memory and I / O performance bottlenecks caused by excessively large single data reads, further improving log download efficiency and overall response speed.

[0091] Optionally, a data prefetching mechanism can be introduced during the block reading process to further reduce I / O waiting time and improve the overall efficiency of log download.

[0092] Specifically, the process of reading redo log files in blocks using an independent download thread and the ASM API interface also includes the following steps:

[0093] Step A: Predict the data blocks that need to be read next based on the currently read AU sequence number.

[0094] During each download thread's block read operation, the system tracks the sequence number of the currently being read allocation unit (AU) in ASM storage in real time. Based on this AU sequence number and the predetermined read order (e.g., incrementing by AU sequence number), combined with the total block count information of the corresponding redo log file, the system dynamically predicts one or more data blocks that need to be read next. For example, if the currently being read data block with AU sequence number 100, and the preset read strategy is sequential read, the system can predict that subsequent data blocks with AU sequence numbers 101, 102, etc., will need to be read. The prediction logic can be configured according to the actual scenario, such as fixed-step prediction or prediction based on access patterns.

[0095] Step B: Based on the predicted data blocks that need to be read later, initiate asynchronous read requests in advance.

[0096] While initiating a synchronous read request for the current Active User (AU), the system asynchronously sends read requests for one or more subsequent AUs to the ASM storage layer based on the prediction results. These asynchronous read requests are submitted through ASM API interfaces (such as the asynchronous mode of `dbms_diskgroup.read`), allowing the storage layer to preload the corresponding data blocks into the memory buffer in the background. When the download thread finishes processing the current data block and prepares to read the next AU, the required data may have already been preloaded into the cache, thus reducing latency caused by waiting for disk I / O operations.

[0097] By prefetching subsequent data blocks, I / O wait time is reduced, avoiding performance bottlenecks caused by reading block by block. This mechanism is particularly effective when reading large files. By preloading subsequent data blocks into the memory cache, it ensures that subsequent read requests do not need to wait for disk I / O, thereby improving overall download efficiency. Furthermore, the combination of asynchronous reading and caching reduces access latency at the storage layer, enabling the system to respond to log download requests more quickly.

[0098] Optionally, this application also provides another specific implementation of the redo log processing method. This method defines in detail the format specifications of the converted local redo log and clarifies the specific process of format conversion to ensure that the output file can be recognized and used by standard analysis tools.

[0099] Specifically, the local redo log format is used to fully describe the structure and content of the redo log file header, including at least the following elements: fixed header identifier, logical block size, number of blocks, and magic value.

[0100] The fixed header identifier is a set of predefined byte sequences used to mark the file type as a redo log; the logical block size defines the standard number of bytes for each logical data block in the file, usually a standard value such as 512 bytes or 1024 bytes; the number of blocks indicates the total number of logical blocks contained in the file; the magic value is a fixed value embedded at a specific position in the header, used as an auxiliary marker to verify and identify the file format version or characteristics.

[0101] The process of converting redo log data read in blocks into local redo log format to obtain local redo logs includes the following steps:

[0102] Step A: Construct a complete file header structure according to the local redo log format specification.

[0103] The system constructs a standard redo log file header in memory according to a predefined format specification. This header contains, in sequence: a predefined fixed header identifier (e.g., sequence 002200000000c0ff000000000000000770b0000), a value indicating the logical block size, a value indicating the total number of blocks in the file, and a magic value located at a predefined position (e.g., 7a7b7c7d). The header length, the offset of each field, and the encoding method all follow the standard redo log file public or parsed specifications.

[0104] Step B: Reassemble the redo log data read in blocks based on the local redo log format to generate a local redo log file.

[0105] The system reads the raw binary data in blocks, using ASM allocation units as units, and then re-cuts, aligns, and concatenates them according to the logical block size to form a continuous sequence of standard logical blocks. Subsequently, the standard file header constructed in step A is sequentially combined with the reassembled data body and written to a new file in the local file system. The final output file is a standardized and structurally complete local redo log, with both its file header and data body conforming to the standard redo log format definition.

[0106] By constructing a standard redo log file header and reassembling data blocks, the compatibility issue between the ASM storage format and standard analysis tools is resolved. The standardized format conversion mechanism ensures the usability of the output files, allowing log files to be directly used by standard analysis tools without additional processing. Furthermore, the design of the header construction and data block reassembly logic reduces the complexity of log conversion, minimizes the need for manual intervention, and improves the accuracy of automated processing.

[0107] Optionally, after completing the log format conversion, this application embodiment also adds a dynamic monitoring and adaptive processing mechanism for changes in the status of the source log file to support seamless downloading from online logs to archived logs.

[0108] Specifically, after converting the redo log data read in chunks into the local redo log format and obtaining the local redo log, the following steps are also included:

[0109] Step A: Detect the status changes of the redo log files in the ASM storage layer.

[0110] The system continuously or periodically monitors the status information of processed redo log files in the ASM storage layer. This monitoring can be achieved by querying metadata related to log status in the database system view, or by directly reading the file's attribute flags through the ASM API interface. The status information is used to distinguish whether the file is currently in an online active state or an archived state. For example, a status of "CURRENT" or "ACTIVE" indicates an online redo log, while a status of "INACTIVE" and a log switch usually indicate that the file has been converted to an archived log.

[0111] Step B: When it is detected that the redo log file in the ASM storage layer has been converted to archive log, automatically switch to archive log download mode.

[0112] When the system detects in step A that the status of a redo log file has changed from online to archived (for example, from "ACTIVE" to "INACTIVE"), and the file path has been moved to the ASM archived log directory structure, a mode switch is automatically triggered. The system will then switch the current or subsequent processing flow for that file from online log download mode to archived log download mode.

[0113] In archived log download mode, the system will reconfigure relevant parameters based on the storage characteristics of the archived logs, for example:

[0114] Update the path matching rules for the target file to point to the directory structure of the archived logs in ASM;

[0115] Adapt to potentially different file naming rules and serial number continuity requirements;

[0116] It can also adjust the read concurrency strategy, buffer size, or retry mechanism as needed to optimize the batch processing efficiency of archived, non-real-time written files.

[0117] After the switch, the system automatically re-initiates a complete or incremental processing flow, including metadata retrieval, chunk reading, and format conversion, based on the new archived log file path and attributes, to ensure that the log data in the archiving stage can be retrieved and converted into the local standard format.

[0118] By dynamically monitoring log file status changes and switching download modes accordingly, the real-time nature of online logs and the integrity of archived logs are ensured. For example, when a log file is converted to archived status, the system can seamlessly switch to archived log download mode, avoiding data loss due to untimely log switching. Furthermore, monitoring log status changes further improves the accuracy of log switch detection, preventing anomalies caused by misjudgments of status.

[0119] Optionally, in the log acquisition process, this application embodiment also introduces a caching mechanism and a dynamic invalidation strategy to reduce the overhead of repeated access to the ASM storage layer and ensure the timeliness of the metadata used.

[0120] Specifically, after converting the redo log data read in chunks into the local redo log format and obtaining the local redo log, the following steps are also included:

[0121] Step A: Temporarily store frequently accessed metadata in the ASM storage layer in the cache.

[0122] The system temporarily stores frequently accessed and relatively stable ASM storage layer metadata in local memory or a distributed cache. Each time metadata is needed, the system first queries the cache; if valid corresponding data exists in the cache, it uses it directly, avoiding repeated calls to the ASM API interface to access the storage layer, thereby reducing network and storage I / O overhead and improving the speed of metadata retrieval.

[0123] Step B: Manage the data in the cache based on the cache invalidation policy.

[0124] To ensure consistency between cached data and the actual state of the ASM storage layer, the system employs a cache invalidation strategy that is linked to database change logic for dynamic cache management. One specific implementation of the cache invalidation strategy is:

[0125] The system employs an invalidation policy based on SCN changes, monitoring the system change number of redo log files associated with cached metadata. When a change in the SCN of a file is detected (e.g., due to log switching, archiving, or new transaction writes causing an update to its associated SCN range), the system automatically marks the cached metadata corresponding to that file as invalid and retrieves the latest data from the ASM storage layer and updates the cache on the next access.

[0126] In addition, cache invalidation strategies can be designed in conjunction with timestamps, polling cycles, or event notification mechanisms to form a multi-dimensional and configurable cache management mechanism.

[0127] Frequently accessed metadata (such as log file path, number of blocks, and block size) is temporarily stored in a cache using a caching mechanism, with a cache invalidation policy set (e.g., based on SCN number changes or time intervals). For example, when the SCN number of a log file is updated, the cache content is automatically refreshed to ensure the timeliness of the metadata. This mechanism can reduce repeated query requests to the ASM storage layer and lower the overhead of storage access.

[0128] To facilitate understanding of the overall process of the redo log processing method in the above embodiments, the following will combine... Figure 2 The overall process of this method will be introduced. Figure 2 This is a schematic diagram illustrating the multi-threaded log reading and switching process provided in an embodiment of this application.

[0129] like Figure 2 As shown, the method mainly includes the following three stages:

[0130] Phase 1: Thread Initialization and Task Dispatch

[0131] After the method starts, it first obtains the thread management information of the current database environment by querying the database system view (e.g., V$THREAD) to identify the various redo threads that need to be processed. Then, it creates an independent download thread for each redo thread and assigns a corresponding starting system change number to each thread as the logical starting point for data reading, thereby ensuring the consistency and order of data across threads while executing concurrently.

[0132] Phase Two: Multi-threaded Block Reading and Localized Writing

[0133] Each download thread executes the following sub-process independently:

[0134] Identify the target log file: Query online or archived log information to locate the specific redo log file that this thread needs to process.

[0135] Get file attributes and open: Call the ASM API interface (e.g., dbms_diskgroup.getfileattr) to get the metadata of the log file (e.g., path, block size, etc.), and then call the interface (e.g., dbms_diskgroup.open) to open the file in the ASM storage layer.

[0136] Construct a standard file header and configure read parameters: Construct a file header containing information such as fixed identifiers and block size according to the standard redo log format, and set the starting offset and data length for this read operation based on the metadata.

[0137] Block reading and local writing: The ASM API read interface (such as dbms_diskgroup.read) is called in a loop to read log data in blocks (e.g., by AU), and the read data is written to the local file system in a standard format.

[0138] Loop and resource release: Determine if the file has been read completely; if not, update the offset and continue reading; if completed, call the close interface (e.g., dbms_diskgroup.close) to release the ASM file handle and update the local block information record.

[0139] Phase 3: Log Status Monitoring and Adaptive Mode Switching

[0140] After a thread finishes reading a log file, it enters the status judgment and subsequent processing stage:

[0141] Check if the log has been switched: Monitor the status of the redo log file in ASM to determine if it has been switched from an online log to an archived log.

[0142] Adaptive path and mode switching:

[0143] If it is detected that the log has not been switched, continue processing the current online log sequence.

[0144] If it is detected that the logs have been switched to archive, the download mode will be automatically switched to archive log download mode, the target file path will be updated to the ASM archive directory, and the process will jump to the next sequence of log files, re-entering the second stage process to achieve continuous log acquisition.

[0145] This flowchart clearly demonstrates how this solution achieves efficient acquisition and conversion of redo logs in ASM storage through multi-threaded concurrency, block reading, standardized reconstruction, and state-aware automatic switching mechanisms, providing reliable technical support for scenarios such as data recovery and audit analysis.

[0146] The electronic device provided in this application embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0147] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the methods in any of the above method embodiments.

[0148] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods in any of the above method embodiments.

[0149] All or part of the steps in the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0150] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0151] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0152] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0153] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

[0154] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0155] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0156] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0157] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0158] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0159] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0160] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A redo log processing method, characterized in that, The method includes: Metadata information of redo log files in the ASM storage layer is obtained through the Automatic Storage Management System (ASM) API interface; wherein, the ASM API interface is used to access the ASM storage layer, the redo log files include log files used to record database change operations, and the metadata information includes at least the file path, number of blocks, block size, and file type; Based on the metadata information, the redo log file is read in blocks through the ASM API interface to obtain the redo log data read in blocks; The redo log data read in blocks is converted into local redo log format to obtain local redo logs.

2. The method according to claim 1, characterized in that, The step of obtaining metadata information of redo log files in the ASM storage layer through the ASM API interface includes: Query the redo thread number in the ASM storage layer; The corresponding starting system change number (SCN) is obtained based on the redo thread number; the starting system change number (SCN) is used to determine the starting position for reading the log file. Access the ASM storage layer through the ASM API interface, and obtain the metadata information corresponding to the redo thread number based on the starting system change number (SCN).

3. The method according to claim 2, characterized in that, The step of reading the redo log file in blocks according to the metadata information through the ASMAPI interface to obtain the redo log data in blocks includes: A separate download thread is assigned to each of the aforementioned redo thread numbers; Based on the independent download thread, the redo log file is read in blocks through the ASM API interface to obtain the redo log data read in blocks, wherein the number of blocks read each time does not exceed a preset number of allocation units (AU).

4. The method according to claim 3, characterized in that, The process of reading the redo log file in blocks through the ASM API interface based on the independent download thread to obtain the redo log data in blocks also includes: Predict the next data blocks to be read based on the currently read AU sequence number; Based on the data blocks that need to be read later, an asynchronous read request is initiated in advance.

5. The method according to any one of claims 1-4, characterized in that, The local redo log format is used to indicate the following contents of the redo log file header: fixed header identifier, logical block size, number of blocks, and magic value; the magic value is used to identify the fixed value of the file format. The process of converting the redo log data read in blocks into the local redo log format to obtain the local redo log includes: The redo log data read in blocks is reassembled based on the local redo log format to obtain the local redo log.

6. The method according to any one of claims 1-4, characterized in that, After converting the redo log data read in blocks into a local redo log format to obtain the local redo log, the process further includes: Detect the status changes of the redo log files in the ASM storage layer; When it is detected that the redo log file in the ASM storage layer has been converted to archive log, the system automatically switches to archive log download mode.

7. The method according to any one of claims 1-4, characterized in that, After converting the redo log data read in blocks into a local redo log format to obtain the local redo log, the process further includes: The frequently accessed metadata in the ASM storage layer is temporarily stored in the cache; The cache invalidation policy manages the source data in the cache; the cache invalidation policy includes: automatically refreshing the contents of the cache when the SCN number of the log file changes.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images