Consistency policy-based cloud-native environment database and application backup and recovery method and system

Abstract

The embodiment of the application relates to a cloud native environment database and application backup recovery method and system based on a consistency strategy, and belongs to the technical field of electronic data processing. The method steps comprise the following: creating a sub-process pool and establishing a bidirectional read-write channel by using a pipe function, realizing inter-process communication and parallel task distribution, determining a storage consistency group in a back-end storage system, calculating a first grade value based on performance monitoring indexes of a source database cluster and generating a recovery verification plan, calculating a second grade value based on the number of data table rows and data volume to screen target data tables, and executing recovery verification in a containerized environment. Through the synergistic cooperation of process-level parallel distribution, storage-level consistency binding and double-level priority screening, the technical problems of inconsistent multi-volume snapshot time points and incomplete recovery verification sampling coverage are solved, and the parallelization and intelligentization of the whole backup recovery process are realized.

Description

Technical Field

[0001] This application relates to a cloud-native environment database and application backup and recovery method and system based on a consistency strategy, belonging to the technical field of electronic data processing. Background Technology

[0002] Currently, database and application backup and recovery in cloud-native environments face multiple technical challenges. Existing technologies typically export databases logically table-by-table in a serial manner. When the database contains hundreds of large tables, the backup window can last for hours or even spans days, severely impacting daily operational scheduling. For containerized applications using multiple persistent volumes, such as database systems that mount data and logs to different storage volumes, existing snapshot solutions can only execute independently for each volume. Due to network jitter or scheduling delays, the snapshot timestamps of different volumes may differ by several seconds, resulting in inconsistencies between the data and log timestamps after recovery, leading to application startup failures. Furthermore, large-scale database backup recovery verification often requires a full recovery, which is costly and time-consuming. Operations teams typically only randomly select a small number of tables for verification, resulting in insufficient verification coverage and failure to detect backup unavailability in a timely manner.

[0003] In engineering practice, the cumulative effect of the above problems is even more pronounced. On the one hand, multi-process backup solutions need to address parallel data consistency between parent and child processes. Without a transaction snapshot synchronization mechanism, different working processes see different data views, leading to timing inconsistencies within the exported logical backup set. On the other hand, backup and recovery verification plans lack dynamic priority scheduling. The verification cycle for high-importance clusters is the same as that for low-importance clusters, resulting in long-term risks to the backup availability of the core system. If only random selection is used during the sampling verification phase, small tables are frequently selected while large tables remain unverified for extended periods. If a large table backup is corrupted, the problem is only discovered during recovery, making it difficult to achieve the business recovery goal. Furthermore, the snapshot generation of multiple associated storage volumes lacks a consistency group management mechanism. Operations personnel need to trigger snapshot operations for each volume at different times, which is difficult to coordinate manually and cannot guarantee strict alignment of snapshot times. Therefore, a solution that integrates multi-volume consistency assurance, parallel backup acceleration, and intelligent verification scheduling is urgently needed. Summary of the Invention

[0004] The main objective of this application is to provide a cloud-native environment database and application backup and recovery method and system based on a consistency strategy, so as to solve the problems of incomplete recovery verification coverage, unreasonable allocation of verification resources, and poor consistency of associated data in the prior art.

[0005] The embodiments of this application adopt the following technical solutions: According to one aspect of the embodiments of this application, a cloud-native environment database and application backup and recovery method based on a consistency strategy is provided, comprising: within a backup controller, creating child processes as a worker process pool by calling the fork function of the main process, and establishing a first read-write channel and a second read-write channel using the pipe function, wherein the first read-write channel retains the main process's read end and the worker process's write end, and the second read-write channel retains the main process's write end and the worker process's read end; in response to the detected consistency group custom resource, obtaining data volume consistency group identification information, and determining the corresponding storage consistency group in the backend storage system according to the data volume consistency group identification information, wherein the storage consistency group is used to bind multiple storage data volumes as logical units; after completing the worker process pool initialization, traversing the data objects to be exported by the main process, and distributing the data object information to idle workers via the second read-write channel. After the worker process completes data export, it returns the results via the first read / write channel. The main process then sets the current worker process's state to idle. Based on the performance monitoring metrics of the source database cluster, the first-level value of each backup database cluster is calculated. A recovery verification plan is generated based on the sorting results. The recovery verification plan is used to instruct the backup data associated with the storage consistency group to be performed for recovery verification. Before executing the recovery verification plan, it is determined whether the amount of backup data in the backup database cluster to be recovered exceeds a preset threshold. The recovery verification type is determined based on the judgment result. If it exceeds the preset threshold, a second-level value is calculated based on the number of rows in the data table and the amount of data. The target data table is selected based on the sorting results, and the recovery verification operation is performed on the backup data in the target data table in a pre-deployed containerized recovery verification environment, thus completing the overall control flow of the backup recovery method.

[0006] According to at least one specific embodiment of the present application, in the backup controller, the process of creating a child process as a worker process pool by calling the fork function of the main process, and establishing a first read-write channel and a second read-write channel using the pipe function, wherein the first read-write channel retains the main process's read end and the worker process's write end, and the second read-write channel retains the main process's write end and the worker process's read end, further includes: the main process initializing a process resource pool and allocating memory for storing worker process information; the main process calling the pipe function to create a pair of pipe communication pipes, returning two pairs of file descriptors, each pair of file descriptors containing a read end and a write end; the main process constructing the first read-write channel and the second read-write channel from the pair of pipe communication pipes, wherein the first read-write channel is formed by the first pipe retaining the main process's read end and the worker process's write end, and the second read-write channel is formed by the second pipe retaining the main process's write end and the worker process's read end; the main process calling the fork function to spawn a child process as a worker process, after the worker process is created, clearing the original database connection object, and opening a new session to reconnect to the database according to the original database connection information.

[0007] According to at least one specific embodiment of the present application, the backup controller is specifically: a containerized control unit running in the backup database cluster, responsible for managing the scheduling, distribution, and monitoring of database backup and recovery tasks; the backup controller internally runs a main process, used to coordinate various worker processes to perform specific data export and import operations; the pipe function is specifically: an inter-process communication function provided by the operating system, used to create a one-way or two-way data transmission channel between the parent process and the child process; the worker process write end is specifically: the end of the worker process writing data into the pipe; the worker process read end is specifically: the end of the worker process reading data from the pipe.

[0008] According to at least one specific embodiment of the present application, the step of obtaining data volume consistency group identification information in response to the detected consistency group custom resource, and determining the corresponding storage consistency group in the backend storage system based on the data volume consistency group identification information, wherein the storage consistency group is used to bind multiple storage data volumes as logical units, further includes: listening to the creation event of the consistency group custom resource in the container cloud platform through the consistency group controller; when a new consistency group custom resource is detected to be created, parsing the identification information of multiple storage data volumes recorded in the custom resource; the consistency group controller calling the application programming interface provided by the backend storage system to create a storage consistency group corresponding to the data volume consistency group in the backend storage system, and adding all storage data volumes recorded in the consistency group custom resource to the storage consistency group.

[0009] According to at least one specific implementation of the embodiments of this application, the consistency group custom resource is specifically: a user-defined backup database cluster resource object, used to declare which storage data volumes need to be bound to a logical group; the data volume consistency group identification information is specifically: a name or number used to uniquely identify a data volume consistency group.

[0010] According to at least one specific embodiment of the present application, the calculation of the first-level value of each backup database cluster based on the performance monitoring indicators of the source database cluster, and the generation of a recovery verification plan based on the sorting results, wherein the recovery verification plan is used for indication, further includes: obtaining the query rate per second indicator value, data volume indicator value, table number indicator value, and CPU utilization indicator value of each source database cluster within a preset time period; matching each indicator value to a pre-divided indicator value interval, obtaining the weight value corresponding to each indicator value interval; for each source database cluster, performing a multiplication operation based on the query rate per second weight value, data volume weight value, table number weight value, and CPU utilization weight value of the source database cluster to calculate the first-level value; and sorting the backup database cluster list in descending order according to the first-level value from high to low to obtain the recovery verification priority.

[0011] According to at least one specific implementation of the embodiments of this application, the data object to be exported is specifically: a database object that needs to be backed up, including: data tables, indexes, views, and stored procedures; the source database cluster is specifically: an original database cluster used to carry business applications and perform daily data read and write operations; the first level value is specifically: a value calculated based on the performance monitoring indicators and weights of the source database cluster.

[0012] According to at least one specific embodiment of the present application, before executing the recovery verification plan, it is determined whether the amount of backup data in the backup database cluster to be recovered exceeds a preset threshold. The recovery verification type is determined based on the judgment result. If it exceeds the preset threshold, a second-level value is calculated based on the number of data table rows and the amount of data. The target data table is filtered according to the sorting result, and the recovery verification operation is performed on the backup data in the target data table in a pre-deployed containerized recovery verification environment to complete the overall control flow of the backup recovery method. The method further includes: obtaining the total storage amount of all backup data in the backup database cluster to be recovered, performing a comparison operation between the total storage amount and a preset data amount threshold. If the total storage amount is less than or equal to the data amount threshold, the recovery verification type is determined to be a full recovery verification type, and full recovery verification is performed on all backup data in the backup database cluster to be recovered; or: if the total storage amount is greater than the data amount threshold, the recovery verification type is determined to be a sampling recovery verification type, and a target data table filtering operation based on the number of data table rows and the amount of data is performed.

[0013] According to at least one specific implementation of the embodiments of this application, the recovery verification plan is specifically: a verification task list generated according to the sorting result of the first level value; the number of rows and the amount of data in the data table are specifically: the number of records contained in the data table and the storage space occupied by the table; the second level value is specifically: a value calculated based on the number of rows and the amount of data in the data table and their respective weights; the containerized recovery verification environment is specifically: an isolated verification environment temporarily built using container technology.

[0014] According to another aspect of the embodiments of this application, a cloud-native environment database and application backup and recovery system based on a consistency strategy is provided to implement the aforementioned cloud-native environment database and application backup and recovery method based on a consistency strategy, comprising: a read-write channel construction module, which, within a backup controller, creates child processes as a worker process pool by calling the fork function of the main process, and establishes a first read-write channel and a second read-write channel using the pipe function, wherein the first read-write channel retains the main process read end and the worker process write end, and the second read-write channel retains the main process write end and the worker process read end; a storage consistency judgment module, which, in response to the detected consistency group custom resources, obtains data volume consistency group identification information, and determines the corresponding storage consistency group in the backend storage system according to the data volume consistency group identification information, wherein the storage consistency group is used to bind multiple storage data volumes as logical units; and a current worker process status configuration module, which, after completing the worker process pool initialization, traverses the data to be exported by the main process. According to the object, the data object information is distributed to the idle worker process via the second read / write channel. After the worker process completes the data export, it returns the output result via the first read / write channel. The main process sets the current worker process status to idle. The backup data execution recovery verification module calculates the first level value of each backup database cluster based on the performance monitoring indicators of the source database cluster, and generates a recovery verification plan based on the sorting result. The recovery verification plan is used to instruct the backup data associated with the storage consistency group to be performed recovery verification. The backup data execution recovery verification operation module determines whether the amount of backup data of the backup database cluster to be recovered exceeds a preset threshold before executing the recovery verification plan. The recovery verification type is determined according to the judgment result. If it exceeds the preset threshold, the second level value is calculated based on the number of data table rows and the amount of data. The target data table is filtered according to the sorting result, and the recovery verification operation is performed on the backup data in the target data table in the pre-deployed containerized recovery verification environment.

[0015] The beneficial technical effects of the embodiments of this application are: This application's embodiments employ a multi-level collaborative control architecture. The main process calls the `fork` function to create a child process pool and uses the `pipe` function to establish first and second read / write channels, constructing a bidirectional, isolated inter-process communication mechanism. Simultaneously, it combines custom resources from consistency groups to bind multiple storage data volumes into logical units. Furthermore, it uses a two-tiered priority ranking system based on a first-level performance monitoring metric and a second-level metric based on the number of rows and the amount of data in the data table. This achieves the integration of parallel backup distribution, multi-volume consistency snapshot binding, and intelligent recovery verification. It realizes the technical effect of collaborative linkage between process communication and status management, storage consistency group binding, and two-tiered priority filtering throughout the backup and recovery process. This solves the technical problems in existing technologies such as low efficiency of serial backup, inconsistent snapshot timestamps across multiple volumes, incomplete recovery verification sampling coverage, excessively long backup windows, failure to restore associated applications, and missing core data verification. Attached Figure Description

[0016] To more clearly illustrate the specific implementation methods of the embodiments of this application or the technical solutions in the prior art, the drawings used in the description of the specific implementation methods or the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a cloud-native environment database and application backup and recovery method based on a consistency strategy.

[0018] Figure 2 This is a flowchart of the preferred technical solution provided in steps S11 to S13.

[0019] Figure 3 This is a flowchart of the preferred technical solution provided in steps S21 to S22.

[0020] Figure 4 This is a flowchart of the preferred technical solution provided in steps S41 to S43.

[0021] Figure 5 This is an architecture diagram of a cloud-native environment database and application backup and recovery system based on a consistency strategy. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the embodiments of this application, and not all embodiments. Based on the specific implementation methods in the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the embodiments of this application.

[0023] like Figure 1 The cloud-native environment database and application backup and recovery method based on consistency strategy shown includes: Step S1: Within the backup controller, the main process calls the fork function to create a child process pool as a worker process pool. The pipe function is used to establish a first read-write channel and a second read-write channel. The first read-write channel retains the main process's read end and the worker process's write end, while the second read-write channel retains the main process's write end and the worker process's read end.

[0024] Step S2: In response to the detected consistency group custom resource, obtain the data volume consistency group identifier information, and determine the corresponding storage consistency group in the backend storage system according to the data volume consistency group identifier information. The storage consistency group is used to bind multiple storage data volumes as logical units.

[0025] Step S3: After the worker process pool is initialized, the main process iterates through the data objects to be exported and distributes the data object information to the idle worker processes through the second read / write channel. After the worker processes complete the data export, they return the results through the first read / write channel, and the main process sets the current worker process to idle.

[0026] Step S4: Calculate the backup data based on the performance monitoring metrics of the source database cluster. The first-level value of the library cluster is used to generate a recovery verification plan based on the sorting results. The recovery verification plan is used to instruct the backup data associated with the storage consistency group to perform recovery verification.

[0027] Step S5: Before executing the recovery verification plan, determine whether the amount of backup data in the backup database cluster to be recovered exceeds a preset threshold. Determine the recovery verification type based on the judgment result. If it exceeds the preset threshold, calculate the second level value based on the number of data table rows and the amount of data. Filter the target data table according to the sorting result, and perform recovery verification operation on the backup data in the target data table in the pre-deployed containerized recovery verification environment to complete the overall control flow of the backup recovery method.

[0028] Steps S1 to S5, through a five-step sequentially connected collaborative control architecture, integrate process-level parallel backup mechanisms, storage-level consistency group binding mechanisms, task distribution and status return mechanisms, priority sorting and verification plan generation mechanisms, and threshold triggering and sampling verification mechanisms. This constructs a fully integrated backup and recovery method from data export to recovery verification, wherein: Step S1 creates a child process pool by calling the fork function of the main process and establishes the first and second read-write channels using the pipe function. This provides an independent inter-process communication infrastructure for the monitoring and processing of custom resources of the consistency group in the subsequent step S2. This ensures that the data volume consistency group identification information obtained in step S2 can be reliably transmitted in the subsequent task distribution process, and solves the technical problem that the control commands cannot be accurately delivered to the working process due to the lack of inter-process communication channels.

[0029] Step S2 defines custom resources based on the monitored consistency groups, determines the corresponding storage consistency groups in the backend storage system, and binds multiple storage data volumes as logical units. Step S3, after initializing the worker process pool, involves the main process iterating through the data objects to be exported and distributing them to idle worker processes via the second read / write channel. After the worker processes complete the export, they return the results via the first read / write channel, and the main process updates the worker process status. The storage consistency groups established in Step S2 provide a basis for defining the scope of the data objects to be exported in Step S3, enabling data from multiple storage data volumes to be included in the same batch of parallel export tasks, preventing the loss of logical coherence when exporting data from multiple volumes separately.

[0030] After data export is completed in step S3, the main process is aware of the execution status and export results of each worker process. Step S4, based on the output of step S3, calculates the first-level value of each backup database cluster using the performance monitoring metrics of the source database cluster, and generates a recovery verification plan based on the sorting results. The unified management of the worker process status by the main process in step S3 provides the prerequisite of task completion for the generation of the recovery verification plan in step S4. This ensures that the verification plan can only be initiated after all data export is complete, preventing unreliable verification results caused by premature verification before the backup data is fully exported.

[0031] The recovery verification plan generated in step S4 instructs the execution of recovery verification on the backup data associated with the storage consistency group. Before executing this plan, step S5 determines whether the amount of backup data in the backup database cluster to be recovered exceeds a preset threshold. Based on the determination result, the recovery verification type is determined. If the threshold is exceeded, a second-level value is calculated based on the number of rows in the data table and the amount of data. Target data tables are then filtered according to the sorting results, and the recovery verification operation is performed in the containerized recovery verification environment. The first-level value verification priority sorting provided in step S4, together with the second-level value sampling screening triggered by the backup data volume threshold in step S5, forms a two-layer filtering mechanism, resolving the issue of excessive resource consumption in full verification and insufficient coverage in random sampling verification.

[0032] like Figure 2As shown, preferably, in step S1, within the backup controller, the main process calls the fork function to create child processes as a worker process pool, and the pipe function is used to establish a first read / write channel and a second read / write channel. The first read / write channel retains the main process's read end and the worker process's write end, while the second read / write channel retains the main process's write end and the worker process's read end. This further includes: Step S11: The main process initializes the process resource pool and allocates memory to store worker process information; the main process calls the pipe function to create a pair of pipe communication pipes, which returns two pairs of file descriptors, each pair of file descriptors containing a read end and a write end.

[0033] In step S12, the main process constructs a first read-write channel and a second read-write channel from the pair of pipe communication channels. The first read-write channel is formed by the first pipe retaining the main process's read end and the worker process's write end, and the second read-write channel is formed by the second pipe retaining the main process's write end and the worker process's read end.

[0034] In step S13, the main process calls the fork function to spawn a child process as a worker process. After the worker process is created, the original database connection objects are cleared, and a new session is started to reconnect to the database based on the original database connection information.

[0035] As an example, in the preferred technical solutions provided in steps S11 to S13: The backup controller is specifically a containerized control unit running in the backup database cluster, which is responsible for managing the scheduling, distribution and monitoring of database backup and recovery tasks. The backup controller has an internal main process that coordinates the various working processes to perform specific data export and import operations.

[0036] The `pipe` function is specifically an inter-process communication function provided by the operating system, used to create a one-way or two-way data transmission channel between a parent process and a child process. Calling the `pipe` function returns two file descriptors, representing the read and write ends of the channel, respectively.

[0037] The worker process write end is specifically the end where the worker process writes data into the pipe. The worker process uses the write end to send execution results, status information, or exported data back to the main process, thus realizing the data flow from the worker process to the main process.

[0038] The worker process read end is specifically the end from which the worker process reads data from the pipe. The worker process receives task instructions and data object information distributed by the main process through the read end, thus realizing the data flow from the main process to the worker process.

[0039] Steps S11 to S13, using the preferred technical solution provided in step S1, involve the main process calling the `pipe` function to create a pair of pipe communication channels, from which a first read-write channel and a second read-write channel are constructed. Specifically, the first pipe retains the main process's read end and the worker process's write end to form the first read-write channel, while the second pipe retains the main process's write end and the worker process's read end to form the second read-write channel. This bidirectional channel construction method, along with the subsequent step S3's "distributing data object information via the second read-write channel and returning results via the first read-write channel," creates a read-write flow binding, resolving the data corruption problem caused by read-write conflicts in single-pipe communication. Simultaneously, after the worker process is created in step S13, the original database connection object is cleared and the database is reconnected, providing an independent database session environment for the parallel export of multiple worker processes in step S3.

[0040] like Figure 3 As shown, preferably, in step S2, in response to the detected consistency group custom resource, data volume consistency group identification information is obtained, and the corresponding storage consistency group is determined in the backend storage system according to the data volume consistency group identification information. The storage consistency group is used to bind multiple storage data volumes as logical units, and further includes: Step S21: Listen for creation events of custom resources of consistency groups in the container cloud platform through the consistency group controller; when a new custom resource of consistency groups is detected to be created, parse the identification information of multiple storage data volumes recorded in the custom resource.

[0041] In step S22, the consistency group controller calls the application programming interface provided by the back-end storage system to create a storage consistency group corresponding to the data volume consistency group in the back-end storage system, and adds all the storage data volumes recorded in the consistency group's custom resources to the storage consistency group.

[0042] As an example, in the preferred technical solution of steps S21 to S22, the consensus group custom resource is specifically a user-defined backup database cluster resource object, used to declare which storage data volumes need to be bound to a logical group. After the controller detects that this resource has been created, it will trigger subsequent consistency snapshot or recovery verification operations. The backup database cluster resource object can be a Kubernetes resource object.

[0043] The data volume consistency group identifier is specifically a name or number used to uniquely identify a data volume consistency group. The main process uses this identifier to find the corresponding consistency group custom resource in the Kubernetes cluster and, based on this, locates the matching storage consistency group in the backend storage system.

[0044] In the preferred technical solution provided in steps S21 to S22, based on step S2, the consistency group controller listens to the creation event of the custom resource of the consistency group, parses the multiple storage data volume identification information recorded therein, and calls the application programming interface of the back-end storage system to create a storage consistency group. This allows the storage consistency group established in step S2 to be directly referenced by the recovery verification plan generated in step S4. That is, the recovery verification plan instructs to perform recovery verification on the backup data associated with the storage consistency group, thus realizing an automated closed loop from resource declaration to back-end storage mapping.

[0045] like Figure 4 As shown, preferably, in step S4, the first-level value of each backup database cluster is calculated based on the performance monitoring indicators of the source database cluster, and a recovery verification plan is generated according to the sorting results. The recovery verification plan is used to indicate that, starting from a preset time, recovery verification is performed on the backup data associated with the storage consistency group, further including: Step S41: Obtain the query rate per second, data volume, number of tables, and CPU utilization of each source database cluster within a preset time period.

[0046] Step S42: Match each indicator value to a pre-divided indicator value range, obtain the weight value corresponding to each indicator value range, and for each source database cluster, perform a multiplication operation based on the query rate per second weight value, data volume weight value, table number weight value and central processing unit utilization weight value of the source database cluster to calculate the first level value.

[0047] Step S43: Sort the backup database cluster list in descending order according to the first-level value from high to low to obtain the recovery verification priority.

[0048] As an example, in the preferred technical solution provided in steps S41 to S43, the data objects to be exported specifically refer to database objects that need to be backed up, including: data tables, indexes, views, and stored procedures. The main process will assign these data objects to be exported one by one to idle worker processes, which will then perform the specific data export operations.

[0049] The source database cluster specifically refers to the original database cluster used to host business applications and perform daily data read and write operations. Backup data originates from the source database cluster, and recovery verification also requires comparison with the data structure and content of the source database cluster.

[0050] The first-level value is specifically calculated based on the performance monitoring metrics and weights of the source database cluster. These performance monitoring metrics include: QPS, data volume, number of tables, CPU utilization, etc. A higher first-level value indicates a more critical database cluster, and its corresponding backup should be prioritized for recovery verification.

[0051] In the preferred technical solutions provided in steps S41 to S43, the weight values ​​of four dimensions—queries per second, data volume, number of tables, and CPU utilization—are obtained, matched with pre-defined intervals of their respective value ranges, and multiplication is performed to calculate the first-level value. The preferred technical solutions provided in steps S41 to S43, based on a multi-dimensional weighted calculation method, provide a priority ranking basis for determining whether the backup data volume of the backup database cluster to be restored and verified exceeds a preset threshold in step S5, enabling the threshold determination in step S5 to prioritize higher-level clusters.

[0052] Preferably, in step S5, before executing the recovery verification plan, it is determined whether the amount of backup data in the backup database cluster to be recovered exceeds a preset threshold. Based on the determination result, the recovery verification type is determined. If it exceeds the preset threshold, a second-level value is calculated based on the number of rows and the amount of data in the data table. Target data tables are then selected according to the sorting results, and recovery verification operations are performed on the backup data in the target data tables within a pre-deployed containerized recovery verification environment, thus completing the overall control flow of the backup recovery method. This further includes: Obtain the total storage volume of all backup data in the backup database cluster to be restored and verified. Compare this total storage volume with a preset data volume threshold. If the total storage volume is less than or equal to the data volume threshold, determine the restoration and verification type as full restoration and verification, and perform full restoration and verification on all backup data in the backup database cluster to be restored and verified. Alternatively, if the total storage volume is greater than the data volume threshold, determine the restoration and verification type as sampling restoration and verification, and perform a target data table filtering operation based on the number of data table rows and the data volume.

[0053] As an example, in the preferred embodiment of step S5, the recovery verification plan specifically comprises a verification task list generated according to the sorting results of the first-level values. The recovery verification plan specifies at what time, for which backup database clusters, and in what order the recovery verification operations will be performed, ensuring that important clusters are verified first.

[0054] The number of rows and the amount of data in the data table specifically refer to the number of records contained in the data table and the amount of storage space occupied by the table. These two metrics are used together to assess the importance of the data table; tables with more rows or larger amounts of data have higher priority during sampling validation.

[0055] The second-level value is specifically calculated based on the number of rows and the amount of data in the table, as well as their respective weights. Tables with more rows or larger amounts of data have a higher second-level value. This value is used to select the highest-priority target data table during sampling recovery verification, prioritizing the verification of the integrity of backup data for core large tables.

[0056] The containerized recovery verification environment is specifically an isolated verification environment temporarily built using container technology. Within this isolated verification environment, backup data can be restored to an independent database instance for integrity verification. After verification, the environment is automatically cleaned up, without affecting the production system.

[0057] In the preferred technical solution of step S5, branch processing of full verification and sampling verification is realized: the preferred technical solution of step S5 compares the total amount of backup data storage with a preset threshold. If it is less than or equal to the threshold, full recovery verification is performed. If it is greater than the threshold, a second-level value is calculated based on the number of rows and the amount of data in the data table, and the target data table is filtered. The preferred technical solution and the first-level value of step S4 form a two-layer filtering mechanism: the first-level value filters out the clusters that need to be verified first, and the second-level value filters out the data tables in the cluster that need to be verified first. Through cooperation, the verification resources are accurately allocated at both the cluster level and the table level.

[0058] like Figure 5 The cloud-native environment database and application backup and recovery system shown is used to implement a cloud-native environment database and application backup and recovery method based on a consistency strategy, including: The read / write channel construction module, within the backup controller, creates child processes as a pool of worker processes by calling the fork function of the main process, and establishes the first and second read / write channels using the pipe function. The first read / write channel retains the read end of the main process and the write end of the worker process, while the second read / write channel retains the write end of the main process and the read end of the worker process.

[0059] The storage consistency judgment module, in response to the detected consistency group custom resource, obtains the data volume consistency group identification information, and determines the corresponding storage consistency group in the backend storage system based on the data volume consistency group identification information. The storage consistency group is used to bind multiple storage data volumes as logical units.

[0060] The current worker process status configuration module, after completing the initialization of the worker process pool, traverses the data objects to be exported through the main process and distributes the data object information to the idle worker processes through the second read / write channel. After the worker process completes the data export, it returns the result through the first read / write channel, and the main process sets the current worker process status to idle.

[0061] The backup data recovery verification module calculates the first-level value of each backup database cluster based on the performance monitoring indicators of the source database cluster, and generates a recovery verification plan according to the sorting results. The recovery verification plan is used to instruct the backup data associated with the storage consistency group to perform recovery verification.

[0062] The backup data recovery verification module determines whether the amount of backup data in the backup database cluster to be recovered exceeds a preset threshold before executing the recovery verification plan. Based on the determination result, the recovery verification type is determined. If it exceeds the preset threshold, a second-level value is calculated based on the number of rows in the data table and the amount of data. The target data table is selected according to the sorting result, and the recovery verification operation is performed on the backup data in the target data table in the pre-deployed containerized recovery verification environment.

[0063] The following section provides a detailed explanation of the cloud-native environment database and application backup and recovery methods based on consistency strategies, using specific application scenarios as examples: In a certain project, a MySQL master-slave replication cluster was deployed as the source database cluster. The cluster consists of 3 nodes, with a total data volume of approximately 2.5TB and about 800 tables. The source database cluster uses a Kubernetes container cloud platform to manage and back up the database cluster. The backup controller runs as a Pod in the cluster and is responsible for scheduling backup tasks. The backend storage uses the Ceph distributed storage system, providing storage consistency group functionality.

[0064] Detailed implementation of step S1: After the backup controller starts, the main process performs initialization operations. The main process first calls the pipe function of the C standard library to create two pipe channels. The underlying implementation of the pipe function is to allocate a memory buffer and return two file descriptors fd[0] (read end) and fd[1] (write end). The first call to pipe gets fd1[0] and fd1[1], and the second call to pipe gets fd2[0] and fd2[1].

[0065] The main process then constructs two read / write channels: the first read / write channel reserves fd1[0] for the main process's read end and fd1[1] for the subsequent child process's write end; the second read / write channel reserves fd2[1] for the main process's write end and fd2[0] for the subsequent child process's read end. After the channel creation is complete, the main process calls the fork function to create the child process. The underlying mechanism of the fork function is to copy the parent process's page table, allocate a new process descriptor for the child process, and the child process obtains a copy of the same code segment, data segment, and file descriptor as the parent process.

[0066] The main process calls `fork` four times consecutively to create four child processes as a worker process pool. After each child process is created, it first closes the file descriptors it does not need: the child process keeps `fd1[1]` (write end) and `fd2[0]` (read end), and closes `fd1[0]` and `fd2[1]`. At the same time, the child process clears the MySQL database connection objects left by the parent process and calls the `mysql_real_connect` function to re-establish the session according to the connection pool configuration, where the connection parameters include IP address 192.168.1.100, port 3306, username backup_user, and password.

[0067] The main process closes fd1[1] and fd2[0] which it does not need, while keeping fd1[0] and fd2[1]. At this point, the bidirectional communication channel is established: the main process sends instructions to the child process's fd2[0] through fd2[1], and the child process sends back the results to the main process's fd1[0] through fd1[1]. The information of the four child processes (PID, file descriptor, current state) is stored in the memory array worker_pool[4] initialized by the main process, and the initial state of all of them is set to IDLE.

[0068] In the specific implementation of step S1, the unidirectional pipe created by the pipe function is essentially a circular buffer in the kernel. Write operations copy data from user space to the kernel buffer, and read operations copy data from the kernel buffer to user space. The bidirectional channel formed by the combination of the two pipes allows parent and child processes to perform read and write operations simultaneously without blocking, because the two pipes correspond to different kernel buffers.

[0069] Detailed implementation of step S2: Users submit a custom ConsistencyGroup resource in YAML format to the Kubernetes cluster using the `kubectl apply` command. The `spec` field of this resource declares the identification information of three storage data volumes: `pvc-db-data-01`, `pvc-db-data-02`, and `pvc-db-log-01`.

[0070] The consistency group controller runs on the same Kubernetes cluster and listens for Add events for ConsistencyGroup resources via the Informer mechanism. When a new resource is detected to be created, the controller parses the resource's metadata and extracts the identifiers of the three data volumes. The controller then calls the Ceph storage system's RBD Python API to perform the following operations: it calls the create_consistency_group interface to create a new storage consistency group, obtaining the group_id as cg-20231105-001; then, for each data volume identifier, it calls the add_image_to_consistency_group interface to add it to the consistency group.

[0071] The underlying implementation of a storage consistency group is as follows: the Ceph backend maintains a common data sequence number for all RBD images within the group. When a snapshot is created for the consistency group, all images within the group are frozen and snapshots are generated at the same point in time, ensuring cross-volume data consistency. Information such as the consistency group's name, UUID, and member volume list is written to the Kubernetes Status field for subsequent steps to read.

[0072] In the specific implementation of step S2, the technical problem addressed by storage consistency groups is that when database data is distributed across multiple persistent volumes, a single-volume snapshot cannot guarantee the point-in-time consistency of data between volumes. Ceph's consistency group implementation mechanism performs a one-time aggregate snapshot operation on all PGs within the group at the OSD level, and the system ensures that all I / O within the group is synchronously paused when the snapshot is created.

[0073] Detailed implementation of step S3: The main process inspects the `worker_pool` array to retrieve worker processes in the IDLE state. Currently, all four processes are in IDLE. The main process then iterates through the list of data objects to be exported. These data objects include metadata for 800 tables: table name, database, storage volume identifier, estimated number of rows, and data volume.

[0074] The main process generates task descriptors for 200 tables in the pvc-db-data-01 data volume. Each descriptor contains a database connection string, table name, export format parameters, and export path. The main process calls the write system function through fd2[1] of the second read / write channel to serialize the task descriptor into a byte stream and write it to the kernel buffer. The corresponding child process reads the byte stream from fd2[0], deserializes it, and then executes the mysqldump command to export the data. The export command format is: mysqldump -h192.168.1.100 -P3306 -ubackup_user db_name table_name --where="1=1" --single-transaction --result-file= / backup / table.sql.

[0075] After the child process completes the export, it checks the exit code of mysqldump. An exit code of 0 indicates success, and a non-zero code indicates failure. The child process writes the execution result (success or failure, number of rows exported, time taken, and error information) back to the kernel buffer through fd1[1] of the first read / write channel. After the main process reads the result from fd1[0], it updates the status of the corresponding process in the worker_pool to IDLE and records the exported result to the log database.

[0076] When a worker process is executing an export task, its status is marked as BUSY, and the master process will not dispatch new tasks to that process. With four worker processes executing in parallel, the total time for exporting 800 tables is reduced from approximately 6 hours for a single process executing serially to approximately 1.5 hours.

[0077] In the specific implementation of step S3, the collaborative mechanism for task distribution and status feedback is based on a producer-consumer model. The main process is the producer, placing tasks into the pipeline queue; the worker processes are the consumers, retrieving tasks from the pipeline for execution. Status management solves the problem of duplicate task distribution: when a worker process fails, its status remains BUSY for an extended period without any results being returned. The main process sets a timeout mechanism (default 30 minutes), and after the timeout, the status is reset to IDLE and the task is redistributed.

[0078] Detailed implementation of step S4: After step S3 completes the data export, the main process retrieves the performance monitoring metrics of the source database cluster over the past 7 days. The monitoring data is stored in Prometheus, and the main process uses PromQL queries to obtain four metric values ​​for each database cluster. Taking cluster A as an example: Queries Per Second (QPS) = 3500, Data Size = 850GB, Table Count = 200, CPU Usage = 65%.

[0079] The following rules apply to the division of indicator value ranges and weight allocation: QPS range (0, 1000) weight 1, (1000, 3000) weight 2, (3000, 5000) weight 3, and above 5000 weight 4; Data volume range (0, 100GB) weight 1, (100GB, 300GB) weight 2, (300GB, 600GB) weight 3, and above 600GB weight 4; Table quantity range (0, 50) weight 1, (50, 150) weight 2, (150, 300) weight 3, and above 300 weight 4; CPU utilization range (0, 30%) weight 1, (30%, 50%) weight 2, (50%, 70%) weight 3, and above 70% weight 4.

[0080] Cluster A has a QPS of 3500, falling within the (3000, 5000) range, with a weight of w_qps = 3; a data volume of 850GB, falling within the 600GB+ range, with a weight of w_data = 4; a number of tables of 200, falling within the (150, 300) range, with a weight of w_table = 3; and a CPU utilization of 65%, falling within the (50%, 70%) range, with a weight of w_cpu = 3. The first-tier score is calculated using multiplication: Score1 = w_qps × w_data × w_table × w_cpu = 3 × 4 × 3 × 3 = 108.

[0081] Performing the same calculation on clusters B and C yielded Score1 values ​​of 72 and 216, respectively. Sorting by Score1 from highest to lowest: cluster C (216) ranked first, cluster A (108) second, and cluster B (72) third. The recovery verification plan was generated based on this sorting result, stipulating that starting at 1:00 AM, the backup data of cluster C would be verified first, followed by clusters A and B in sequence. The plan also included the verification window duration for each cluster (30 minutes per cluster by default) and the storage consistency group identifier cg-20231105-001.

[0082] In the specific implementation of step S4, multiplication of the first-level values ​​amplifies the differences between metrics more effectively than addition. When a particular metric of the cluster is exceptionally high, the product increases significantly, giving it a higher ranking. The four metrics correspond to business importance (QPS reflects access frequency), scale (data volume and number of tables reflect recovery complexity), and load pressure (CPU utilization reflects cluster activity; verification during off-peak periods can reduce the impact on business). This ranking calculation method is based on the weighted product model in multi-attribute decision theory.

[0083] Detailed implementation of step S5: Before executing the recovery verification plan, the total backup data storage of the cluster C to be verified was determined to be 720GB. The preset data volume threshold is 500GB; 720GB is greater than 500GB, therefore the recovery verification type is determined to be sampling recovery verification. The row count and data volume of all tables in cluster C were obtained: the largest order table, order_line, contains 230 million rows and 85GB of data; the second largest, order table, contains 45 million rows and 18GB of data. The second-tier score is calculated using the normalized product of Score2 = (row_count / 10000) × (data_size / 1GB). After sorting and filtering, the top 10% of target data tables were selected, and backup data recovery and integrity verification were performed in a containerized verification environment (using Docker to start a temporary MySQL instance).

[0084] Definitions: The pipe function: A circular buffer managed by the operating system kernel, providing a channel for byte stream transmission.

[0085] The fork function copies the parent process's page table using copy-on-write technology, giving the child process an independent address space.

[0086] Consistent group custom resources: an extension of the Kubernetes declarative API, providing a description of the state that users expect.

[0087] Storage Consistency Group: A group management unit at the storage system level that creates synchronized snapshots of multiple volumes.

[0088] First-level value multiplication operation: a weighted product model in multi-attribute decision-making, used to amplify the differences between multiple indicators.

[0089] Work process status management: Task queues and worker status tables in the producer-consumer model.

[0090] Furthermore, the technical solutions of the various implementation methods in this application can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the embodiments of this application.

[0091] All features disclosed in the embodiments of this application, or all steps in the disclosed methods or processes, may be combined in any way, except for mutually exclusive features and / or steps. Any feature disclosed in the specification of the embodiments of this application, unless specifically stated otherwise, may be replaced by other equivalent or similar alternative features. That is, unless specifically stated otherwise, each feature is merely one example of a series of equivalent or similar features. Throughout the specification, the same reference numerals indicate the same elements.

[0092] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification of embodiments (including the corresponding claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification of embodiments (including the corresponding claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of this application, and are not intended to limit them. Although the embodiments of this application have been described in detail with reference to the foregoing specific embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing specific embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions provided by the specific embodiments of this application.

Claims

1. A cloud-native environment database and application backup and recovery method based on a consistency strategy, characterized in that, include: Within the backup controller, the main process calls the fork function to create child processes as a pool of worker processes. The pipe function is used to establish a first read-write channel and a second read-write channel. The first read-write channel retains the main process's read end and the worker process's write end, while the second read-write channel retains the main process's write end and the worker process's read end. In response to the detected consistency group custom resource, the data volume consistency group identification information is obtained, and the corresponding storage consistency group is determined in the backend storage system based on the data volume consistency group identification information. The storage consistency group is used to bind multiple storage data volumes as logical units. After the worker process pool is initialized, the main process iterates through the data objects to be exported and distributes the data object information to the idle worker processes through the second read / write channel. After the worker processes complete the data export, they return the results through the first read / write channel, and the main process sets the current worker process to idle. The first-level value of each backup database cluster is calculated based on the performance monitoring indicators of the source database cluster. A recovery verification plan is generated based on the ranking results. The recovery verification plan is used to instruct the backup data associated with the storage consistency group to perform recovery verification. Before executing the recovery verification plan, it is determined whether the amount of backup data in the backup database cluster to be recovered exceeds a preset threshold. The recovery verification type is determined based on the judgment result. If it exceeds the preset threshold, a second-level value is calculated based on the number of rows in the data table and the amount of data. The target data table is selected according to the sorting result, and the recovery verification operation is performed on the backup data in the target data table in the pre-deployed containerized recovery verification environment, thus completing the overall control flow of the backup recovery method.

2. The cloud-native environment database and application backup and recovery method based on consistency strategy according to claim 1, characterized in that, Within the backup controller, the main process calls the `fork` function to create child processes as a worker process pool. The `pipe` function establishes a first read / write channel and a second read / write channel. The first read / write channel retains the main process's read end and the worker process's write end, while the second read / write channel retains the main process's write end and the worker process's read end. This further includes: The main process initializes the process resource pool and allocates memory to store worker process information; the main process calls the pipe function to create a pair of pipe communication pipes, which returns two pairs of file descriptors, each pair of file descriptors containing a read end and a write end; The main process constructs a first read-write channel and a second read-write channel from the pair of pipe communication pipes. The first read-write channel is formed by the first pipe retaining the main process's read end and the worker process's write end, and the second read-write channel is formed by the second pipe retaining the main process's write end and the worker process's read end. The main process calls the fork function to spawn a child process as a worker process. After the worker process is created, it clears the original database connection objects and starts a new session to reconnect to the database based on the original database connection information.

3. The cloud-native environment database and application backup and recovery method based on consistency strategy according to claim 2, characterized in that, The backup controller is specifically a containerized control unit running in the backup database cluster, which is responsible for managing the scheduling, distribution and monitoring of database backup and recovery tasks. The backup controller has an internal main process that coordinates the various working processes to perform specific data export and import operations. The pipe function is specifically an inter-process communication function provided by the operating system, used to create a one-way or two-way data transmission channel between a parent process and a child process. The worker process writing end is specifically the end where the worker process writes data into the pipe. The working process read end is specifically the end from which the working process reads data from the pipe.

4. The cloud-native environment database and application backup and recovery method based on consistency strategy according to claim 1, characterized in that, In response to the detected consistency group custom resource, data volume consistency group identification information is obtained, and the corresponding storage consistency group is determined in the backend storage system based on the data volume consistency group identification information. The storage consistency group is used to bind multiple storage data volumes as logical units, and further includes: The consistency group controller listens for creation events of custom consistency group resources in the container cloud platform; when a new custom consistency group resource is detected to be created, the identification information of multiple storage data volumes recorded in the custom resource is parsed. The consistency group controller calls the application programming interface provided by the backend storage system to create a storage consistency group corresponding to the data volume consistency group in the backend storage system, and adds all storage data volumes recorded in the consistency group's custom resources to the storage consistency group.

5. The cloud-native environment database and application backup and recovery method based on consistency strategy according to claim 4, characterized in that, The consistency group custom resource is specifically a user-defined backup database cluster resource object, used to declare which storage data volumes need to be bound to a logical group; The data volume consistency group identification information specifically refers to the name or number used to uniquely identify a data volume consistency group.

6. The cloud-native environment database and application backup and recovery method based on consistency strategy according to claim 1, characterized in that, The first-level value of each backup database cluster is calculated based on the performance monitoring indicators of the source database cluster, and a recovery verification plan is generated according to the ranking result. The recovery verification plan is used to indicate and further includes: Obtain the query rate per second, data volume, number of tables, and CPU utilization for each source database cluster within a preset time period. Each indicator value is matched to a pre-divided indicator value range, and the weight value corresponding to each indicator value range is obtained. For each source database cluster, a multiplication operation is performed based on the query rate per second weight value, data volume weight value, table number weight value and central processing unit utilization weight value of the source database cluster to calculate the first level value. The backup database cluster list is sorted in descending order according to the first-level value from high to low to obtain the recovery verification priority.

7. The cloud-native environment database and application backup and recovery method based on consistency strategy according to claim 6, characterized in that, The data objects to be exported are specifically database objects that need to be backed up, including: data tables, indexes, views, and stored procedures. The source database cluster is specifically: the original database cluster used to host business applications and perform daily data read and write operations; The first level value is specifically calculated based on the performance monitoring indicators and weights of the source database cluster.

8. The cloud-native environment database and application backup and recovery method based on consistency strategy according to claim 1, characterized in that, Before executing the recovery verification plan, it is determined whether the amount of backup data in the backup database cluster to be recovered exceeds a preset threshold. Based on the determination result, the recovery verification type is determined. If it exceeds the preset threshold, a second-level value is calculated based on the number of rows in the data table and the amount of data. Target data tables are selected according to the sorting results, and recovery verification operations are performed on the backup data in the target data tables in a pre-deployed containerized recovery verification environment, thus completing the overall control flow of the backup recovery method. This further includes: Obtain the total storage volume of all backup data in the backup database cluster to be restored and verified. Compare this total storage volume with a preset data volume threshold. If the total storage volume is less than or equal to the data volume threshold, determine the restoration and verification type as a full restoration and verification type. Perform a full restoration and verification on all backup data in the backup database cluster to be restored and verified, or: If the total storage volume exceeds the data volume threshold, the recovery verification type is determined to be the sampling recovery verification type, and a target data table filtering operation based on the number of data table rows and the data volume is performed.

9. The cloud-native environment database and application backup and recovery method based on consistency strategy according to claim 8, characterized in that, The recovery verification plan is specifically a verification task list generated according to the sorting results of the first-level values; The number of rows and the amount of data in the data table specifically refer to the number of records contained in the data table and the amount of storage space occupied by the table. The second level value is specifically calculated based on the number of rows and the amount of data in the data table, as well as their respective weights. The containerized recovery verification environment specifically refers to an isolated verification environment temporarily built using container technology.

10. A cloud-native environment database and application backup and recovery system based on a consistency strategy, used to implement the cloud-native environment database and application backup and recovery method based on a consistency strategy as described in any one of claims 1 to 9, characterized in that, include: The read / write channel construction module, within the backup controller, creates child processes as a pool of worker processes by calling the fork function of the main process, and establishes the first and second read / write channels using the pipe function. The first read / write channel retains the read end of the main process and the write end of the worker process, while the second read / write channel retains the write end of the main process and the read end of the worker process. The storage consistency judgment module, in response to the detected consistency group custom resource, obtains the data volume consistency group identification information, and determines the corresponding storage consistency group in the backend storage system based on the data volume consistency group identification information. The storage consistency group is used to bind multiple storage data volumes as logical units. The current worker process status configuration module, after completing the initialization of the worker process pool, traverses the data objects to be exported through the main process and distributes the data object information to the idle worker processes through the second read / write channel. After the worker process completes the data export, it returns the result through the first read / write channel, and the main process sets the current worker process status to idle. The backup data recovery verification module calculates the first-level value of each backup database cluster based on the performance monitoring indicators of the source database cluster, and generates a recovery verification plan according to the sorting results. The recovery verification plan is used to instruct the backup data associated with the storage consistency group to perform recovery verification. The backup data recovery verification module determines whether the amount of backup data in the backup database cluster to be recovered exceeds a preset threshold before executing the recovery verification plan. Based on the determination result, the recovery verification type is determined. If it exceeds the preset threshold, a second-level value is calculated based on the number of rows in the data table and the amount of data. The target data table is selected according to the sorting result, and the recovery verification operation is performed on the backup data in the target data table in the pre-deployed containerized recovery verification environment.

Images