Database standby machine incremental reconstruction idempotent retry method, device and electronic equipment

Abstract

The application relates to a database backup machine incremental reconstruction idempotent retry method and device. The method comprises the following steps: dividing a reconstruction process into three stages of pre-preparation, intermediate transmission and post-closing, and allowing retry only in the first two stages; first calculating and persisting a master-standby common WAL consistency site common-checkpoint and a difference structure filemap, then sequentially copying a WAL, a serialized change-block mirror and a filemaptarget to a disk; after fault recovery, reverse serializing and continuing transmission, realizing incremental level idempotent retry; if the error context is the same for two times in succession, immediately converging. Through accurate retry control and state persistence, the application significantly reduces the full amount construction triggering probability, improves the data consistency guarantee capability, simultaneously has the non-invasive integration advantage, and can effectively improve the availability and operation and maintenance efficiency of a database system.

Description

Technical Field

[0001] This application relates to the field of database operation technology, and in particular to a database standby incremental reconstruction idempotent retry method, apparatus, computer-readable storage medium, and electronic device. Background Technology

[0002] In distributed database systems, the master-slave architecture is one of the core technical means to ensure high availability and data reliability. Typically, the master database (host) handles real-time write operations and generates transaction logs (such as redo logs), while the slave database (standby machine) achieves data consistency by synchronizing the transaction logs of the master database, allowing for rapid service failover in the event of a master database failure. The data synchronization process on the standby machine mainly consists of two modes: full build and incremental build. A full build refers to the standby machine completely fetching and loading basic data from the master database, and then synchronizing subsequent logs. This mode consumes significant network bandwidth, computing resources, and time, potentially leading to prolonged standby machine unavailability. Incremental build, on the other hand, is based on the existing synchronization state of the standby machine (such as already synchronized log positions), only transmitting missing log fragments to catch up on data, offering advantages such as low resource consumption and short recovery time. Therefore, reducing the probability of triggering a full build is crucial for improving the availability and efficiency of the database system.

[0003] However, in actual operation, the standby machine is often forced to interrupt synchronization due to network disconnection, abnormal restart of the primary database, or primary / standby role switching, requiring a restart of the data reconstruction process. In such abnormal scenarios, directly triggering a full build will significantly increase system resource consumption and service recovery time. To address this, existing technologies attempt to introduce a retry mechanism for incremental builds, allowing the standby machine to re-initiate incremental synchronization in the aforementioned abnormal scenarios to avoid full builds as much as possible. However, this mechanism still has key drawbacks: on the one hand, the retry strategy lacks specificity and does not fully consider the root causes of incremental build failures in abnormal scenarios (such as temporary interruptions caused by network jitter or log position offsets after the primary database restarts), which may exacerbate resource waste due to blind retries; on the other hand, the repeated retry mechanism does not distinguish between error types. If the error messages (such as "log position mismatch" or "connection timeout") are exactly the same in two consecutive attempts, invalid retries will still be initiated, which may eventually force a rollback to a full build, failing to effectively reduce the actual probability of triggering a full build.

[0004] In summary, how to achieve intelligent retries for incremental builds in abnormal scenarios such as network disconnection, primary database restart, and primary / standby switchover—supporting necessary retries to improve synchronization success rate while avoiding invalid retries through error message identification—has become an urgent technical problem to be solved. Summary of the Invention

[0005] To address the aforementioned issues, this application proposes a novel idempotent retry method for incremental database standby rebuilds. This invention aims to address the pain points mentioned above by designing an idempotent retry method that, through precise control of retry conditions, significantly reduces the probability of triggering a full build during standby rebuilds, thereby improving the overall availability of the database system.

[0006] To achieve the above objectives, this application provides the following technical solution: The first aspect of this application provides an idempotent retry method for incremental reconstruction of a database standby machine, the method comprising: S1. On the standby side, the incremental reconstruction process is divided into a pre-preparation stage, an intermediate data transmission stage, and a post-closing stage. Retry is only allowed in the pre-preparation stage and the intermediate data transmission stage, while retry is prohibited in the post-closing stage. S2. In the pre-preparation stage, establish a connection with the primary database, calculate the latest common WAL consistency point (common-checkpoint) between the primary and standby databases, and generate a primary-standby difference structure (filemap) based on this point. If this stage fails due to network interruption, primary database restart, or primary-standby switchover, the serialization results of the common-checkpoint and filemap are saved to the local disk and the process exits for direct loading during the next retry. S3. After entering the intermediate data transmission stage, the following steps are executed sequentially: S31. Copy WAL data to the current master database based on the persistent common-checkpoint, and serialize and write the change-block set parsed from the WAL data to disk in real time to form a change-block image file; S32. Based on the filemap and change-block image files, perform copy, remove, truncate and create operations on the standby local data files, and serialize and write the metadata filemaptarget to be written to the target file to disk before each operation. S33. Before writing the first data file, generate an atomic marker file build_complete.start locally to indicate that the intermediate data transfer stage has been entered; S4. When the intermediate data transmission phase is interrupted due to network disconnection, CRC check failure, or full disk space, only the files written by this end during the current retry cycle are deleted, while the common-checkpoint, change-block image file, and filemaptarget are retained. After the fault is recovered, the above information is deserialized from the disk, and S3 is re-executed to achieve idempotent retry of incremental reconstruction. S5. If the error code and error context captured in two consecutive retries are completely identical, terminate the retries and report the failure to prevent an invalid retry storm. S6. Once all filemap operations are complete and WAL replication catches up with the master database, the post-completion stage begins. The atomic marker file is renamed, backup_label is written, and pg_control is updated. At this point, retries are disabled, and incremental reconstruction ends.

[0007] Furthermore, in the method of this application, the common-checkpoint needs to perform CRC consistency verification with the master database before each retry. If the verification fails, it is determined that a master-slave switch has occurred, and the copied old master data is immediately cleaned up and the difference structure is recalculated.

[0008] Furthermore, in the method of this application, both the change-block image file and the filemaptarget file are stored in the $PGDATA / gs_build_retry / directory on the standby machine and are atomically deleted after the incremental reconstruction is successful.

[0009] Furthermore, in the method of this application, the filemaptarget only records the file path, size and LSN range on the standby side, and does not save the actual page content, so as to ensure that the persistence overhead is less than 1% of the incremental data volume.

[0010] Furthermore, in the method of this application, the "deleting the file written by this end during the current retry cycle" in step S4 is achieved by comparing the file modification time with the current retry timestamp, ensuring that the page that was successfully written in the previous retry is not destroyed.

[0011] Furthermore, in the method of this application, the existence of the atomic marker file build_complete.start is used as the state boundary between the pre-preparation stage and the intermediate data transmission stage. If the existence of this file is detected when the system starts up and pg_control is not updated, the retry process is automatically initiated.

[0012] Furthermore, in the method of this application, the error context includes the main library SystemID, Timeline, error code, failed file name and offset. Only when all of the above fields are consistent is it determined to be "the same error".

[0013] A second aspect of this application provides a database standby incremental reconstruction idempotent retry apparatus, the apparatus comprising: The phase division module is used to divide the incremental reconstruction process into a pre-preparation phase, an intermediate data transmission phase, and a post-closing phase on the standby side. Retry is allowed only in the pre-preparation phase and the intermediate data transmission phase, while retry is prohibited in the post-closing phase. The pre-preparation stage processing module is used to establish a connection with the primary database, calculate the latest common WAL consistency point between the primary and standby databases (common-checkpoint), and generate a primary-standby difference structure filemap based on the point. If this stage fails due to network interruption, primary database restart, or primary-standby switchover, the serialization results of the common-checkpoint and filemap are retained to the local disk before exiting, so that they can be directly loaded during the next retry. The intermediate data transmission stage processing module is used to execute the following processes sequentially after entering the intermediate data transmission stage: (1) Copy WAL data to the current master database based on the persistent common-checkpoint, and serialize the change-block set parsed from the WAL data to disk in real time to form a change-block image file; (2) Based on the filemap and change-block image files, perform copy, remove, truncate and create operations on the standby local data files, and serialize and write the metadata filemaptarget to be written to the target file to disk before each operation; (3) Before writing the first data file, generate an atomic marker file build_complete.start locally to indicate that the intermediate data transfer stage has been entered; The idempotent retry control module is used to delete only the files written by the local end during the current retry cycle when the intermediate data transmission phase is interrupted due to network disconnection, CRC check failure, or full disk space, while retaining the common-checkpoint, change-block image file, and filemaptarget; after the fault is recovered, the above information is deserialized from the disk and the data file operation is re-executed to achieve idempotent retry for incremental reconstruction. The error convergence module is used to terminate the retry and report the failure when the error code and error context captured in two consecutive retries are completely identical, thus preventing an invalid retry storm. The post-completion stage processing module is used to rename the atomic marker file, write backup_label, and update pg_control after all filemap operations are completed and WAL replication catches up with the master database. At this point, retries are prohibited, and incremental reconstruction ends.

[0014] The device implements the steps of the aforementioned database standby incremental reconstruction idempotent retry method during operation.

[0015] Furthermore, the device of this application also includes a primary / standby switchover detection module, which is used to perform CRC consistency verification on the common-checkpoint before retrying. If the verification fails, it triggers the recalculation of the difference structure and the cleanup of the old primary data.

[0016] Furthermore, the device of this application is integrated within the database instance process, and provides incremental reconstruction idempotent retry capability without interface changes through the command gs_ctl build -D $PGDATA -b incremental.

[0017] A third aspect of this application provides an electronic device, including: a memory and a processor; Memory: Used to store computer programs; Processor: Used to execute the computer program to implement the steps of the aforementioned database standby incremental reconstruction idempotent retry method.

[0018] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the aforementioned database standby incremental reconstruction idempotent retry method.

[0019] In summary, this invention, through a phased idempotent retry mechanism and a critical state persistence design, significantly improves the success rate and efficiency of standby machine reconstruction at a lower cost, and has the following technical advantages: (1) Precise control in stages → effectively reduce the probability of full construction Incremental rebuilding is explicitly divided into three phases: preparation, transmission, and finalization, with differentiated retry strategies (such as allowing retries for specific errors only during the preparation and transmission phases) to avoid invalid operations. This directly enables the system to intelligently retry incremental synchronization in scenarios such as network anomalies or master database restarts, reducing the probability of triggering a full build by approximately 30%-40% and significantly saving network and computing resources.

[0020] (2) State persistence and data hedging → ensuring strong data consistency Serialize and store key information such as WAL change blocks, LSN synchronization points, and file mapping tables (see [link]). Figure 3 , Figure 4 ( ), the state can be directly restored when retrying.

[0021] By combining data offsetting after primary / standby switchover (deleting old primary data + resynchronizing), we ensure that data remains consistent even after the primary / standby role switchover, avoiding logical errors caused by data forks.

[0022] (3) Error detection and process optimization → Improve system stability and efficiency By comparing consecutive error messages to block invalid retries, and combining this with serial writing and periodic fsync operations, redundant I / O is reduced. This increases the retry success rate to over 85% and reduces reconstruction time by approximately 40%, while also preventing system load spikes caused by repeated retries.

[0023] (4) Non-intrusive integration → Enables low-cost deployment Integration via standard command (gs_ctl build -b auto / incremental) requires no modification to the database kernel or interface. This advantage allows the solution to seamlessly adapt to existing database systems, significantly reducing deployment and maintenance costs, and is particularly suitable for distributed environments with high availability requirements.

[0024] In summary, this invention, through its three core advantages of precise retry control, state persistence, and lightweight integration, directly translates into three beneficial effects: reducing the risk of full-scale construction, ensuring data consistency, and optimizing resource utilization efficiency. It has become a key technical solution for improving the reliability and availability of database systems.

[0025] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings involved in the description of the embodiments will be briefly introduced below. It should be noted that the accompanying drawings only show some embodiments of this application. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0027] Figure 1 The overall implementation flowchart of the idempotent retry method for incremental reconstruction of the database standby machine in this application is shown.

[0028] Figure 2 This is a schematic diagram of the data flow between the primary and backup machines during incremental reconstruction in an embodiment of this application.

[0029] Figure 3 This is a schematic diagram of an idempotent retry scheme in a non-master / backup switchover scenario in an embodiment of this application.

[0030] Figure 4 This is a schematic diagram of an idempotent retry scheme for a primary / backup switchover scenario in an embodiment of this application.

[0031] Figure 5 This is a structural diagram of the idempotent retry device for incremental reconstruction of the database standby machine in this application.

[0032] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0034] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0035] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0036] Figure 1 The diagram shows the overall implementation flow of the database standby incremental reconstruction idempotent retry method provided in this application, including the following steps: S1. On the standby side, the incremental reconstruction process is divided into a pre-preparation stage, an intermediate data transmission stage, and a post-closing stage. Retry is only allowed in the pre-preparation stage and the intermediate data transmission stage, while retry is prohibited in the post-closing stage. S2. In the pre-preparation stage, establish a connection with the primary database, calculate the latest common WAL consistency point (common-checkpoint) between the primary and standby databases, and generate a primary-standby difference structure (filemap) based on this point. If this stage fails due to network interruption, primary database restart, or primary-standby switchover, the serialization results of the common-checkpoint and filemap are saved to the local disk and the process exits for direct loading during the next retry. S3. After entering the intermediate data transmission stage, the following steps are executed sequentially: S31. Copy WAL data to the current master database based on the persistent common-checkpoint, and serialize and write the change-block set parsed from the WAL data to disk in real time to form a change-block image file; S32. Based on the filemap and change-block image files, perform copy, remove, truncate and create operations on the standby local data files, and serialize and write the metadata filemaptarget to be written to the target file to disk before each operation. S33. Before writing the first data file, generate an atomic marker file build_complete.start locally to indicate that the intermediate data transfer stage has been entered; S4. When the intermediate data transmission phase is interrupted due to network disconnection, CRC check failure, or full disk space, only the files written by this end during the current retry cycle are deleted, while the common-checkpoint, change-block image file, and filemaptarget are retained. After the fault is recovered, the above information is deserialized from the disk, and S3 is re-executed to achieve idempotent retry of incremental reconstruction. S5. If the error code and error context captured in two consecutive retries are completely identical, terminate the retries and report the failure to prevent an invalid retry storm. S6. Once all filemap operations are complete and WAL replication catches up with the master database, the post-completion stage begins. The atomic marker file is renamed, backup_label is written, and pg_control is updated. At this point, retries are disabled, and incremental reconstruction ends.

[0037] To more clearly illustrate the technical solution of this application, the following will provide further explanation through specific scenario embodiments.

[0038] Based on a thorough understanding of the changes made to local files at each stage of incremental build and the impact of these changes, this application logically divides the incremental backup process into the following stages: 1. Preparatory Stage The pre-processing phase primarily involves finding the primary database (host) and generating a filemap to represent the differences between the primary and backup data. In summary, it mainly involves comparing local and source data to generate the primary / backup data difference structure and locate WAL consistency checkpoints. No backup data is written during this phase. In this phase, excluding scenarios such as local file corruption, no commonalities between local and source WAL records, or inconsistencies in systemid and timeline between local and source servers, retry is possible.

[0039] In addition, at this stage, the incremental to full conversion caused by the inability to connect to the master is the most meaningful (especially in the second stage when the network connection is interrupted and the reconnection to the master fails, while the marked file exists).

[0040] 2. Intermediate data transmission stage During the intermediate data transmission phase, there are two main processes: WAL (Write-Ahead Log) transmission and data transmission. The main reasons for anomalies here may include insufficient disk space, network disconnection (network disconnection is the most common), and CRC (Corrected Logic Registry) failure in sending data (mainly caused by source CRC verification failure; several such cases have been encountered on the live network, caused by frequent updates of a certain page leading to multiple CRC consistency checks failing).

[0041] This stage generates the most critical status file, build_compete.start. Currently, the mechanism is such that incremental builds are no longer supported when this file is found in the first stage.

[0042] Furthermore, because the data transfer is in progress, local files, including WAL and data files, have already begun to be modified. The modification logic for the WAL file is to perform fsync when the file is found to be full, or when the WAL transfer is completed. The data file is modified by retrieving the content of the file to be modified through SQL, then writing it serially through the global file descriptor, and finally performing a unified fsync.

[0043] 3. Post-construction finishing stage This stage involves renaming the tag file, recording the backup_label file, and updating the pg_controldata file, and is generally error-free. Once this stage is entered, because it involves modifying pg_control and other cleanup operations, retries are no longer allowed.

[0044] To ensure idempotent retries, it's necessary to guarantee that the data retrieved from the main source is identical each time. Currently, the transmission is segmented, comprising three parts of data: 1. The primary focus of master WAL data synchronization is determining the starting point from which master WAL data begins to be synchronized. This starting point is called the common checkpoint, which is the latest consistency point of the WAL between the master and backup.

[0045] 2. Block change data based on backup WAL data statistics.

[0046] 3. After comparing the primary and backup data, file-level operations such as copy and remove may be required.

[0047] Specific data flow is as follows: Figure 2 As shown: The first and second parts are calculated based on WAL; the third part is obtained by comparing the data directories between the primary and backup systems.

[0048] Based on the above analysis, to achieve idempotent retries, the data acquired by the WAL logs must first be backed up (written to disk). Currently, the logic overwrites the backed-up WAL file. Based on this conclusion, the change-block information generated by the WAL logs needs to be backed up (by serializing it from memory to disk). During retries, this information should be deserialized from disk back to memory to mitigate the impact of the WAL file being overwritten. Simultaneously, relevant LSN information (consistency sites) needs to be stored to mark the starting point for copying the WAL logs to the master database, such as... Figure 3 As shown.

[0049] The above solution primarily addresses the block data modified by the standby machine's unique WAL (Write-Ahead Log) and the log synchronization start point. In addition, there's data synchronization. Because the primary machine's data path changes, previously generated data file differences were compared with the primary machine's historical data. To ensure accuracy, it now needs to be compared with the primary machine's real-time data. However, the standby machine's data directory has been overwritten and is unreliable. Therefore, only the standby machine's data file information can be recorded and persisted (this information is saved in filemaptarget), such as... Figure 4 As shown.

[0050] However, considering the possibility of a master-slave switch and the probability of data fork between the new and old masters, CRC consistency verification needs to be added to the cached common-checkpoint. Simultaneously, since there's a high probability of copying data from the old master, and this data cannot be guaranteed to be consistent with the new master in terms of content, data hedging is implemented from the following two perspectives: 1. Data based on the old master copy needs to be deleted.

[0051] 2. Data needs to be copied from the new master to overwrite the data deleted from the old master.

[0052] The incremental reconstruction idempotent retry function described in this embodiment does not require any changes to the existing interface; the command remains: gs_ctl build -D $PGDATA -b auto / incremental.

[0053] Figure 5 The image shows a database standby incremental reconstruction idempotent retry device proposed in this application, the device comprising: The phase division module is used to divide the incremental reconstruction process into a pre-preparation phase, an intermediate data transmission phase, and a post-closing phase on the standby side. Retry is allowed only in the pre-preparation phase and the intermediate data transmission phase, while retry is prohibited in the post-closing phase. The pre-preparation stage processing module is used to establish a connection with the primary database, calculate the latest common WAL consistency point between the primary and standby databases (common-checkpoint), and generate a primary-standby difference structure filemap based on the point. If this stage fails due to network interruption, primary database restart, or primary-standby switchover, the serialization results of the common-checkpoint and filemap are retained to the local disk before exiting, so that they can be directly loaded during the next retry. The intermediate data transmission stage processing module is used to execute the following processes sequentially after entering the intermediate data transmission stage: (1) Copy WAL data to the current master database based on the persistent common-checkpoint, and serialize the change-block set parsed from the WAL data to disk in real time to form a change-block image file; (2) Based on the filemap and change-block image files, perform copy, remove, truncate and create operations on the standby local data files, and serialize and write the metadata filemaptarget to be written to the target file to disk before each operation; (3) Before writing the first data file, generate an atomic marker file build_complete.start locally to indicate that the intermediate data transfer stage has been entered; The idempotent retry control module is used to delete only the files written by the local end during the current retry cycle when the intermediate data transmission phase is interrupted due to network disconnection, CRC check failure, or full disk space, while retaining the common-checkpoint, change-block image file, and filemaptarget; after the fault is recovered, the above information is deserialized from the disk and the data file operation is re-executed to achieve idempotent retry for incremental reconstruction. The error convergence module is used to terminate the retry and report the failure when the error code and error context captured in two consecutive retries are completely identical, thus preventing an invalid retry storm. The post-completion stage processing module is used to rename the atomic marker file, write backup_label, and update pg_control after all filemap operations are completed and WAL replication catches up with the master database. At this point, retries are prohibited, and incremental reconstruction ends.

[0054] When the above-mentioned device is in operation, it implements the steps of the database standby incremental reconstruction idempotent retry method disclosed in this application.

[0055] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of apparatus, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0056] like Figure 6 As shown in the illustration, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the aforementioned database standby incremental reconstruction idempotent retry method.

[0057] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0058] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types: Object-oriented programming languages, such as Java, Smalltalk, C++, etc. Conventional procedural programming languages, such as "C" or similar programming languages.

[0059] The execution methods of program code include, but are not limited to: It runs entirely on the user's computer; Part of it executes on the user's computer, and part of it executes on a remote computer; Execute as a standalone software package; It is executed entirely on a remote computer or server.

[0060] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.

[0061] Furthermore, this application also discloses a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the various steps of the database standby incremental reconstruction idempotent retry method disclosed in this application.

[0062] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following: (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0063] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0064] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0065] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0066] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0067] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0068] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0069] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0070] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0071] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0072] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0073] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program includes program code for executing the database standby incremental reconstruction idempotent retry method disclosed in this application. When the computer program is executed by a processing device, it can achieve the functions defined in the embodiments of this application.

[0074] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0075] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A database standby incremental reconstruction idempotent retry method, characterized in that, The method includes: S1. On the standby side, the incremental reconstruction process is divided into a pre-preparation stage, an intermediate data transmission stage, and a post-closing stage. Retry is only allowed in the pre-preparation stage and the intermediate data transmission stage, while retry is prohibited in the post-closing stage. S2. In the pre-preparation stage, establish a connection with the primary database, calculate the latest common WAL consistency point (common-checkpoint) between the primary and standby databases, and generate a primary-standby difference structure (filemap) based on this point. If this stage fails due to network interruption, primary database restart, or primary-standby switchover, the serialization results of the common-checkpoint and filemap are saved to the local disk and the process exits for direct loading during the next retry. S3. After entering the intermediate data transmission stage, the following steps are executed sequentially: S31. Copy WAL data to the current master database based on the persistent common-checkpoint, and serialize and write the change-block set parsed from the WAL data to disk in real time to form a change-block image file; S32. Based on the filemap and change-block image files, perform copy, remove, truncate and create operations on the standby local data files, and serialize and write the metadata filemaptarget to be written to the target file to disk before each operation. S33. Before writing the first data file, generate an atomic marker file build_complete.start locally to indicate that the intermediate data transfer stage has been entered; S4. When the intermediate data transmission phase is interrupted due to network disconnection, CRC verification failure, or full disk space, only the files written by this end during the current retry cycle are deleted, while the common-checkpoint, change-block image file, and filemaptarget are retained. After the fault is recovered, the common-checkpoint, change-block image file, and filemaptarget are deserialized from the disk, and S3 is re-executed to achieve idempotent retry of incremental reconstruction. S5. If the error code and error context captured in two consecutive retries are completely identical, terminate the retries and report the failure to prevent an invalid retry storm. S6. Once all filemap operations are complete and WAL replication catches up with the master database, the post-completion stage begins. The atomic marker file is renamed, backup_label is written, and pg_control is updated. At this point, retries are disabled, and incremental reconstruction ends.

2. The method according to claim 1, characterized in that, Before each retry, the common-checkpoint needs to perform CRC consistency verification with the master database. If the verification fails, it is determined that a master-slave switch has occurred, and the copied old master data is immediately cleaned up and the difference structure is recalculated.

3. The method according to claim 1, characterized in that, Both the change-block image file and the filemaptarget file are stored in the $PGDATA / gs_build_retry / directory on the standby machine and are atomically deleted after the incremental reconstruction is successful.

4. The method according to claim 1, characterized in that, The filemaptarget only records the file path, size, and LSN range on the standby machine side, and does not save the actual page content, so as to ensure that the persistence overhead is less than 1% of the incremental data volume.

5. The method according to claim 1, characterized in that, The step S4, "deleting files written by this end within this retry cycle," is achieved by comparing the file modification time with the current retry timestamp, ensuring that pages successfully written in the previous retry are not destroyed.

6. The method according to claim 1, characterized in that, The existence of the atomic marker file build_complete.start is used as the state boundary between the pre-preparation stage and the intermediate data transmission stage. If the system detects the existence of this file when it starts up and pg_control has not been updated, it will automatically enter the retry process.

7. The method according to claim 1, characterized in that, The error context includes the main database SystemID, Timeline, error code, failed file name, and offset. Only when all of the above error contexts are consistent is it determined to be "the same error".

8. A database standby incremental reconstruction idempotent retry device, characterized in that, include: The phase division module is used to divide the incremental reconstruction process into a pre-preparation phase, an intermediate data transmission phase, and a post-closing phase on the standby side. Retry is allowed only in the pre-preparation phase and the intermediate data transmission phase, while retry is prohibited in the post-closing phase. The pre-preparation stage processing module is used to establish a connection with the primary database, calculate the latest common WAL consistency point between the primary and standby databases (common-checkpoint), and generate a primary-standby difference structure filemap based on the point. If this stage fails due to network interruption, primary database restart, or primary-standby switchover, the serialization results of the common-checkpoint and filemap are retained to the local disk before exiting, so that they can be directly loaded during the next retry. The intermediate data transmission stage processing module is used to execute the following processes sequentially after entering the intermediate data transmission stage: (1) Copy WAL data to the current master database based on the persistent common-checkpoint, and serialize the change-block set parsed from the WAL data to disk in real time to form a change-block image file; (2) Based on the filemap and change-block image files, perform copy, remove, truncate and create operations on the standby local data files, and serialize and write the metadata filemaptarget to be written to the target file to disk before each operation; (3) Before writing the first data file, generate an atomic marker file build_complete.start locally to indicate that the intermediate data transfer stage has been entered; The idempotent retry control module is used to delete only the files written by the local end during the current retry cycle when the intermediate data transmission phase is interrupted due to network disconnection, CRC check failure, or full disk space, while retaining the common-checkpoint, change-block image file, and filemaptarget. After the fault is recovered, the common-checkpoint, change-block image file, and filemaptarget are deserialized from the disk, and the intermediate data transmission phase processing module is re-executed to achieve idempotent retry for incremental reconstruction. The error convergence module is used to terminate the retry and report the failure when the error code and error context captured in two consecutive retries are completely identical, thus preventing an invalid retry storm. The post-completion phase processing module is used to enter the post-completion phase after all filemap operations are completed and WAL replication catches up with the master database. It renames the atomic marker file, writes backup_label and updates pg_control. At this point, retries are prohibited and incremental reconstruction ends.

9. The apparatus according to claim 8, characterized in that, The device is integrated within the database instance process and provides incremental reconstruction idempotent retry capability without interface changes through the command gs_ctl build -D $PGDATA -b incremental.

10. An electronic device, characterized in that, include: Memory and processor; Memory: Used to store computer programs; Processor: Used to execute the computer program to implement the steps of the database standby incremental reconstruction idempotent retry method as described in any one of claims 1-7.

Images