Big data-based relational database backup recovery method and system

Abstract

The invention provides a relational database backup recovery method and system based on big data, and relates to the technical field of database backup recovery, and the method comprises the following steps: obtaining total backup data and incremental backup data of a relational database, and obtaining merged backup data; based on the features and the access mode of the merged backup data, constructing a multi-dimensional index structure at the same time, and integrating a data dependency graph, a classification result and cold and hot layering information into a multi-dimensional index; decompressing the target backup data to obtain recovered backup data, and performing intelligent verification on the recovered backup data by using a data integrity constraint rule; if the intelligent verification is passed, the recovered backup data is imported into the relational database, and data recovery at a specified time point is completed; and if the intelligent verification is not passed, intelligent data restoration and complementation are carried out, and then verification, verification and importing are carried out again until the intelligent verification is passed.

Description

Technical Field

[0001] The present invention relates to database backup and recovery technology, and in particular to a relational database backup and recovery method and system based on big data. Background Art

[0002] With the continuous growth of data volume and the increasing complexity of business systems, the backup and recovery of relational databases are facing many challenges. Traditional backup and recovery methods usually use a combination of full backup and incremental backup. However, when the backup data volume is large, the data structure is complex, and the data is highly correlated, the data merging, compression, storage, and recovery process is time-consuming and inefficient, and it is difficult to meet the needs of rapid recovery to a specified time point. In addition, the high storage cost of backup data, high data redundancy, and difficulty in ensuring data integrity are also issues that need to be addressed.

[0003] In order to meet the above challenges, there is an urgent need for a relational database backup and recovery method and system based on big data technology to achieve intelligent classification, automatic merging, rapid positioning and intelligent verification of backup data, improve the efficiency, reliability and flexibility of data backup and recovery, reduce storage costs, and ensure data integrity and consistency. Summary of the invention

[0004] The embodiments of the present invention provide a big data-based relational database backup and recovery method and system, which can solve the problems in the prior art.

[0005] According to a first aspect of the embodiments of the present invention, A big data-based relational database backup and recovery method is provided, comprising: Obtain the full backup data and incremental backup data of the relational database, use the machine learning algorithm to intelligently classify the full backup data and incremental backup data, and build a data dependency graph based on the classification results, analyze the correlation and integrity constraints between the data, use the graph neural network to analyze the data dependency graph, identify changes in the data structure, and automatically generate a data merging strategy based on the identification results. Use a parallel computing framework to intelligently merge the full backup data and incremental backup data according to the data merging strategy to obtain the merged backup data; Based on the characteristics and access patterns of the merged backup data, a compression algorithm combination is automatically selected using an adaptive optimization algorithm, the merged backup data is hierarchically compressed to obtain compressed backup data, the compressed backup data is stored in a distributed storage system, and data block distribution is optimized, life cycle prediction is performed on the compressed backup data stored in the distributed storage system, cold and hot stratification of data is achieved, and a multidimensional index structure is constructed at the same time, and the data dependency graph, classification results, and cold and hot stratification information are integrated into the multidimensional index; When it is necessary to restore the relational database to a specified time point, the multidimensional index structure is used to quickly locate the target backup data, and the parallel decompression technology corresponding to the compression algorithm combination is used to decompress the target backup data to obtain the restored backup data, and the restored backup data is intelligently verified using data integrity constraint rules; if the intelligent verification passes, the restored backup data is imported into the relational database to complete the data recovery at the specified time point; if the intelligent verification fails, intelligent data repair and completion are performed, and then verification, verification and import are performed again until the intelligent verification passes.

[0006] In an optional embodiment, Based on the classification results, a data dependency graph is constructed to analyze the correlation and integrity constraints between data. The data dependency graph is analyzed using a graph neural network to identify changes in data structure, and a data merging strategy is automatically generated based on the identification results. A parallel computing framework is used to intelligently merge the full backup data and incremental backup data according to the data merging strategy. The merged backup data includes: Based on the classification results, identify the primary key and unique key fields of each table, compare these fields one-to-one with all the fields of all other tables in the database, identify potential foreign key relationships by comparing the similarity of field names, data types, lengths, and value ranges, calculate the data overlap and distribution similarity between the identified potential related fields, and determine the strength of correlation between the fields; Based on the identified foreign key relationships and field correlation strengths, a data dependency graph is constructed, where nodes represent data tables and edges represent relationships between tables. The constructed data dependency graph is input into the graph neural network model, and through multi-layer graph convolution and pooling operations combined with the attention mechanism, an embedding vector for each data table node is generated; Calculate the cosine similarity between the embedding vector of the current timestamp node and the embedding vector of the historical timestamp node, judge the changes in the data table according to the preset similarity threshold, identify newly added tables, tables with data volume growth exceeding the preset threshold, and large-scale data update events, and generate data structure change recognition results; According to the results of data structure change identification, a data merging strategy is automatically generated. Based on the data merging strategy, the backup data is divided into independent processing units according to the data table, and a directed acyclic graph of the merging task is constructed. At the same time, the distributed computing framework is utilized, and the fault tolerance and retry mechanism are set up to perform parallel merging operations and generate the final merged backup data.

[0007] In an optional embodiment, Construct a directed acyclic graph of the merge task, use the distributed computing framework, set up fault tolerance and retry mechanisms, perform parallel merge operations, and generate the final merged backup data including: In the distributed computing framework, according to the pre-set serverless computing model, the data merging task is split into multiple independent functions, where each function corresponds to a node in the directed acyclic graph. Each function is deployed to the serverless platform, and the function trigger configuration is generated according to the edge relationship in the directed acyclic graph. And the task scheduling plan is generated according to the priority of function execution; According to the task scheduling plan and function trigger configuration, the serverless platform is integrated with the message queue service to create a data transmission channel. The object storage service of the serverless platform is used to establish a result repository for storing intermediate results of function execution. The monitoring tool of the serverless platform is used to collect the number of function calls, execution time, memory usage and error rate, and generate a performance report. Based on the performance report and the directed acyclic graph structure, a function optimization configuration including memory allocation and timeout time is created; According to the function optimization configuration, the number of retries and error handling rules for function execution are set, a fault-tolerant solution is generated, and a parallel processing framework for simultaneously executing multiple non-dependency functions is created by utilizing the distributed characteristics of the serverless platform and the parallel execution path of the directed acyclic graph; Based on the parallel processing framework, function optimization configuration and fault-tolerant scheme, functions are executed in the order of priority in the task scheduling plan. After the functions corresponding to all nodes of the directed acyclic graph are executed, the final merging function is triggered. The final merging function reads all intermediate results, integrates them and generates the final merged backup data.

[0008] In an optional embodiment, The life cycle of compressed backup data stored in the distributed storage system is predicted to achieve hot and cold data stratification, and a multi-dimensional index structure is constructed, including: Perform multi-dimensional life cycle analysis on the compressed backup data to obtain the historical access frequency and time distribution information of the data blocks, and build a data life cycle prediction model based on the historical access frequency and time distribution information; Using the data lifecycle prediction model, predicting the future access frequency and time distribution of each data block, setting a multi-level hot and cold data judgment threshold according to the predicted future access frequency and time distribution, and classifying each data block into a hot data block, a warm data block, and a cold data block based on the multi-level hot and cold data judgment threshold; Construct a data dependency graph to record the associations between the data blocks, allocate hot data blocks to high-performance storage devices, allocate warm data blocks to medium-performance storage devices, and allocate cold data blocks to low-cost storage devices. According to the classification results and storage locations of each data block, formulate a hierarchical storage optimization strategy including data pre-fetching, cache management, and data migration; Execute the tiered storage optimization strategy, record the access frequency, access delay and migration times of each data block, generate a performance optimization report, and determine the index dimensions and weights of the multidimensional index based on the performance optimization report; From the compressed backup data, metadata information is extracted according to the index dimensions and weights to generate an initial metadata set, which is preprocessed to obtain structured metadata. Based on the data distribution characteristics and query patterns of the structured metadata, performance indicators of different multidimensional index structures are calculated. Based on the performance indicators, the optimal multidimensional index structure is selected as the final multidimensional index structure.

[0009] In an optional embodiment, Based on the data distribution characteristics and query patterns of the structured metadata, the formula for calculating the performance indicators of different multidimensional index structures is as follows: ; Among them, M represents the multidimensional index performance index, P represents the number of different multidimensional indexes, and R j A represents the query response efficiency of the j-th multidimensional index. j represents the storage space occupied by the j-th multidimensional index, S j represents the number of dimensions of the jth index, γ represents the weight coefficient, δ represents the attenuation factor, and D j represents the update frequency of the jth index, Q represents the number of different queries, and T k Indicates the processing time of the kth query.

[0010] In an optional embodiment, If the intelligent verification fails, intelligent data repair and completion will be performed, including: If the intelligent verification fails, intelligent data repair is performed, and the errors in the data are automatically corrected using the integrity constraint rules to obtain preliminary repaired data. The integrity constraint rules are converted into machine-executable repair operations, and the constraint satisfaction problem solving algorithm is used to infer a repair solution that satisfies the constraint conditions. According to the repair solution, the search direction of the repair algorithm is guided to further optimize the preliminary repaired data to obtain the target repaired data; Apply the association rule mining algorithm to the target repair data to obtain the dependency relationship between attributes. According to the dependency relationship between attributes and the values ​​of known attributes, the values ​​of missing attributes are inferred to obtain preliminary completed data. The initially completed data is represented as a high-dimensional sparse matrix, a matrix decomposition algorithm is used to obtain potential patterns and similarities in the data, and a factor matrix is ​​obtained. The factor matrix is ​​used to calculate missing values, and the initially completed data is secondary completed according to the missing values. A collaborative filtering algorithm is used to predict the missing numerical types in the secondary completed data according to historical data of similar users or projects, and the collaborative filtering completed data is obtained. According to the collaborative filtering completion data obtained, a deep learning-based generative model is trained, and the trained generative model is used to generate missing attribute values ​​to obtain the generative model completion data. The completed data is represented as a graph structure, and graph neural network modeling and learning are used. Through message passing and aggregation operations, the existence probability of missing node attributes or edges is inferred, and finally the completed graph structure data is obtained to achieve intelligent data repair and completion.

[0011] In an optional embodiment, The method further comprises: Use causal discovery algorithms to learn the causal relationship between variables from data, build a causal graph model, determine the parent node and child node of the missing value node based on the constructed causal graph, and establish the dependency relationship between the missing value and the causally related variables; Using the conditional probability distribution model, the conditional probability distribution of the missing value node is calculated when the parent node value is given, and the preliminary repair and completion results of the missing value are generated according to the estimated conditional probability distribution; Construct corresponding simulation scenarios for the preliminary repair and completion results. In the simulation scenarios, design adaptive model selection and parameter tuning mechanisms, automatically select causal discovery algorithms and causal effect estimation methods according to data characteristics and task requirements, and use causal graph models and the selected causal effect estimation methods to estimate the causal effects of the preliminary repair and completion results on key variables; According to the estimated results of the causal effect, the repair and completion schemes are optimized, and the scheme with the best causal effect is selected as the final repair and completion result.

[0012] According to a second aspect of the embodiments of the present invention, Provided is a relational database backup and recovery system based on big data, comprising: The first unit is used to obtain full backup data and incremental backup data of a relational database, intelligently classify the full backup data and incremental backup data using a machine learning algorithm, and build a data dependency graph based on the classification results, analyze the correlation and integrity constraints between data, use a graph neural network to analyze the data dependency graph, identify changes in data structure, and automatically generate a data merging strategy based on the identification results, adopt a parallel computing framework, and intelligently merge the full backup data and incremental backup data according to the data merging strategy to obtain merged backup data; The second unit is used to automatically select a compression algorithm combination based on the characteristics and access mode of the merged backup data using an adaptive optimization algorithm, perform hierarchical compression on the merged backup data to obtain compressed backup data, store the compressed backup data in a distributed storage system, optimize data block distribution, perform life cycle prediction on the compressed backup data stored in the distributed storage system, realize hot and cold stratification of data, and construct a multidimensional index structure to integrate the data dependency graph, classification results, and hot and cold stratification information into the multidimensional index; The third unit is used to use a multidimensional index structure to quickly locate target backup data when it is necessary to restore the relational database to a specified time point, and to decompress the target backup data using a parallel decompression technology corresponding to the compression algorithm combination to obtain restored backup data, and to perform intelligent verification on the restored backup data using data integrity constraint rules; if the intelligent verification passes, the restored backup data is imported into the relational database to complete data recovery at the specified time point; if the intelligent verification fails, intelligent data repair and completion are performed, and then verification, verification and import are performed again until the intelligent verification passes.

[0013] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0014] According to a fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0015] In this embodiment, the backup data is intelligently classified by machine learning algorithms, and the data dependency is analyzed by graph neural networks to automatically generate the optimal data merging strategy. The parallel computing framework is combined to achieve efficient data merging and processing, significantly improving the efficiency of backup recovery. The adaptive optimization algorithm is used to automatically select the optimal compression algorithm combination, and the merged backup data is hierarchically compressed to effectively reduce the data storage space occupied. The data block optimization distribution of the distributed storage system is combined to further reduce the storage cost. A multidimensional index structure is constructed to integrate the data dependency graph, classification results, and hot and cold stratification information into the index to achieve rapid positioning and access to backup data. The parallel decompression technology corresponding to the compression algorithm is used to accelerate the data recovery process. The restored backup data is intelligently verified using data integrity constraint rules to ensure the correctness and consistency of the restored data. In the case of failure to pass the verification, intelligent data repair and completion technology is used to automatically repair the data to ensure data integrity. Based on the access mode and characteristics of the data, the backup data is predicted for life cycle and hot and cold data is stratified to optimize data storage and access strategies. Support recovery to a specified time point to meet data recovery requirements in different business scenarios. The distributed storage architecture supports horizontal expansion and elastic scaling of backup data to adapt to the growing amount of data and high concurrent access requirements. The parallel computing framework and multi-dimensional index structure enable high scalability of the backup and recovery process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a process flow of a relational database backup and recovery method based on big data according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a relational database backup and recovery system based on big data according to an embodiment of the present invention; Figure 3 It is a framework diagram of the scheme of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0019] Figure 1FIG. 1 is a flow chart of a method for backing up and restoring a relational database based on big data according to an embodiment of the present invention. Figure 1 As shown, the method includes: S101. Obtain the full backup data and incremental backup data of the relational database, use a machine learning algorithm to intelligently classify the full backup data and the incremental backup data, and build a data dependency graph based on the classification results, analyze the correlation and integrity constraints between the data, use a graph neural network to analyze the data dependency graph, identify changes in the data structure, and automatically generate a data merging strategy based on the identification results. Use a parallel computing framework to intelligently merge the full backup data and the incremental backup data according to the data merging strategy to obtain the merged backup data.

[0020] In an optional embodiment, Based on the classification results, a data dependency graph is constructed to analyze the correlation and integrity constraints between data. The data dependency graph is analyzed using a graph neural network to identify changes in data structure, and a data merging strategy is automatically generated based on the identification results. A parallel computing framework is used to intelligently merge the full backup data and incremental backup data according to the data merging strategy. The merged backup data includes: Based on the classification results, identify the primary key and unique key fields of each table, compare these fields one-to-one with all the fields of all other tables in the database, identify potential foreign key relationships by comparing the similarity of field names, data types, lengths, and value ranges, calculate the data overlap and distribution similarity between the identified potential related fields, and determine the strength of correlation between the fields; Based on the identified foreign key relationships and field correlation strengths, a data dependency graph is constructed, where nodes represent data tables and edges represent relationships between tables. The constructed data dependency graph is input into the graph neural network model, and through multi-layer graph convolution and pooling operations combined with the attention mechanism, an embedding vector for each data table node is generated; Calculate the cosine similarity between the embedding vector of the current timestamp node and the embedding vector of the historical timestamp node, judge the changes in the data table according to the preset similarity threshold, identify newly added tables, tables with data volume growth exceeding the preset threshold, and large-scale data update events, and generate data structure change recognition results; According to the results of data structure change identification, a data merging strategy is automatically generated. Based on the data merging strategy, the backup data is divided into independent processing units according to the data table, and a directed acyclic graph of the merging task is constructed. At the same time, the distributed computing framework is utilized, and the fault tolerance and retry mechanism are set up to perform parallel merging operations and generate the final merged backup data.

[0021] Exemplarily, firstly, by analyzing the metadata information of the data table, the association relationship between the tables is identified, and a data dependency graph is constructed. Among them, by comparing the similarity of field name, data type, length and value range, potential foreign key relationships are identified, and the data overlap and distribution similarity between the fields are calculated to determine the correlation strength between the fields. The nodes of the data dependency graph represent data tables, and the edges represent the relationship between the tables, which reflects the dependency and association relationship between the tables in the database.

[0022] Next, the constructed data dependency graph is input into the graph neural network model for processing. Graph neural network is a deep learning model specifically designed to process graph structured data. By encoding and propagating the nodes and edges in the graph, the relationship between nodes and the feature representation of nodes can be learned. In this method, multi-layer graph convolution and pooling operations are used to extract features and reduce the dimension of the data dependency graph. The neighborhood information of the node is aggregated through the convolution operation, and the feature compression and abstraction are achieved through the pooling operation. At the same time, the attention mechanism is introduced to adaptively adjust the weights of node features according to the correlation and importance between nodes, highlighting the influence of key nodes. After multi-layer graph convolution and pooling operations, as well as the support of the attention mechanism, an embedding vector for each data table node is finally generated, indicating the position and role of the table in the data dependency graph.

[0023] In order to identify changes in data structures, the concept of timestamp is introduced. By comparing the cosine similarity between the embedding vector of the current timestamp node and the embedding vector of the historical timestamp node, it can be determined whether the data table has changed significantly. Cosine similarity is a commonly used vector similarity measurement method, which measures the similarity between two vectors by calculating the cosine value of the angle between them. If the cosine similarity between the embedding vector of the current timestamp node and the embedding vector of the historical timestamp node is lower than the preset threshold, it is considered that the data table has changed significantly. By setting a suitable similarity threshold, the sensitivity of change detection can be flexibly controlled. Based on the results of the similarity comparison, newly added tables, tables with data volume growth exceeding the preset threshold, and events of large-scale data updates can be identified to generate data structure change identification results.

[0024] Automatically generate data merge strategies based on the results of data structure change identification. Data merge strategies refer to how to merge backup data with different timestamps to ensure the integrity and consistency of the merged data. By analyzing the dependencies and changes between data tables, the order and method of merge operations can be determined. For example, for newly added tables, they need to be directly added to the merged data set; for tables whose data volume growth exceeds the preset threshold, the incremental data needs to be merged with the original data; for tables with large-scale data updates, the new data version needs to replace the original data version.

[0025] When executing the data merge operation, a parallel processing strategy is adopted to divide the backup data into independent processing units according to the data table, and build a directed acyclic graph (DAG) of the merge task. Each data table corresponds to a merge task, and the dependencies between tasks are determined according to the association between the data tables. By building the DAG of the merge task, the execution order and dependencies of the tasks can be clarified, ensuring the correctness and efficiency of the merge operation.

[0026] In order to speed up the data merging process, this method uses a distributed computing framework to assign the merging task to multiple computing nodes for parallel execution. Distributed computing frameworks such as Apache Spark and Apache Flink provide convenient data parallel processing and task scheduling functions, which can make full use of cluster resources and improve the efficiency of merging operations. At the same time, in order to ensure the reliability of the merging operation, this method sets up a fault tolerance and retry mechanism. When executing the merging task, if there are abnormal situations such as node failure or network interruption, the system will automatically reschedule the failed task to other available computing nodes for retry until the task is successfully completed. Through the fault tolerance and retry mechanism, the success rate of the merging operation can be maximized to avoid data loss or inconsistency.

[0027] In this embodiment, the association relationship between data tables is automatically identified, a data dependency graph is constructed, and the structural characteristics of the database are accurately characterized. The data dependency graph is learned using a graph neural network to generate an embedding vector representing the characteristics of each data table, providing support for subsequent change identification and merging strategy generation. By comparing the embedding vectors of table nodes with different timestamps, the changes in the data structure are identified, including newly added tables, tables with a sharp increase in data volume, and large-scale data update events, etc., providing a basis for formulating merging strategies. The data merging strategy is automatically generated, and the order and method of the merging operation are determined according to the changes and dependencies of the data tables to ensure the integrity and consistency of the merged data. The parallel processing and distributed computing framework are adopted to improve the efficiency and scalability of data merging, and can handle large-scale backup data sets. Fault tolerance and retry mechanisms are set to ensure the reliability of the merging operation and avoid data loss or inconsistency problems caused by abnormal situations.

[0028] In an optional embodiment, Construct a directed acyclic graph of the merge task, use the distributed computing framework, set up fault tolerance and retry mechanisms, perform parallel merge operations, and generate the final merged backup data including: In the distributed computing framework, according to the pre-set serverless computing model, the data merging task is split into multiple independent functions, where each function corresponds to a node in the directed acyclic graph. Each function is deployed to the serverless platform, and the function trigger configuration is generated according to the edge relationship in the directed acyclic graph. And the task scheduling plan is generated according to the priority of function execution; According to the task scheduling plan and function trigger configuration, the serverless platform is integrated with the message queue service to create a data transmission channel. The object storage service of the serverless platform is used to establish a result repository for storing intermediate results of function execution. The monitoring tool of the serverless platform is used to collect the number of function calls, execution time, memory usage and error rate, and generate a performance report. Based on the performance report and the directed acyclic graph structure, a function optimization configuration including memory allocation and timeout time is created; According to the function optimization configuration, the number of retries and error handling rules for function execution are set, a fault-tolerant solution is generated, and a parallel processing framework for simultaneously executing multiple non-dependency functions is created by utilizing the distributed characteristics of the serverless platform and the parallel execution path of the directed acyclic graph; Based on the parallel processing framework, function optimization configuration and fault-tolerant scheme, functions are executed in the order of priority in the task scheduling plan. After the functions corresponding to all nodes of the directed acyclic graph are executed, the final merging function is triggered. The final merging function reads all intermediate results, integrates them and generates the final merged backup data.

[0029] For example, first, according to the pre-set serverless computing model, the data merging task is divided into several independent functions. Each function corresponds to a node in the directed acyclic graph, representing an atomic operation or processing step in the data merging process. By reasonably dividing the granularity and responsibilities of the functions, the modularization and decoupling of the tasks can be achieved, and the flexibility and maintainability of the system can be improved.

[0030] Next, deploy the divided functions to the serverless computing platform. The serverless computing platform is a computing model based on event-driven and function-as-a-service (FaaS). It allows developers to encapsulate application logic into independent functions, and the platform automatically manages the execution environment and resource allocation of the functions. By deploying functions to the serverless platform, automatic scaling and on-demand execution of functions can be achieved, reducing the system's operation and maintenance costs and management complexity.

[0031] When deploying a function, a function trigger configuration is generated based on the edge relationship in the directed acyclic graph. The function trigger configuration defines the calling relationship and trigger conditions between functions to ensure that the functions are executed in the correct order and dependency relationship. By analyzing the topological structure of the directed acyclic graph, the function trigger configuration can be automatically generated, simplifying the workload of configuration management.

[0032] In order to optimize the execution efficiency and resource utilization of functions, this method introduces a task scheduling plan. The task scheduling plan determines the scheduling order and execution time of functions according to the execution priority and dependency of functions. By reasonably arranging the task scheduling plan, the concurrency and throughput of functions can be maximized, and the waiting time and resource waste of tasks can be reduced.

[0033] When actually executing the data merging task, this method uses the message queue service and object storage service provided by the serverless platform to build a data transmission channel and result repository. The message queue service is used to transfer data and trigger signals between functions to ensure the reliable flow of data between functions. The object storage service is used to store the intermediate results generated during the function execution process, providing high-availability and high-durability data storage capabilities. By integrating the message queue and object storage service of the serverless platform, data sharing and collaboration between functions can be achieved, improving the efficiency and reliability of data merging.

[0034] In order to monitor and optimize the execution performance of the function, this method uses the monitoring tools provided by the serverless platform to collect indicators such as the number of function calls, execution time, memory usage, and error rate, and generate a performance report. The performance report reflects the actual execution status and resource consumption of the function, providing a basis for optimizing the function configuration. According to the structural characteristics of the performance report and the directed acyclic graph, a function optimization configuration including memory allocation and timeout can be created, and the resource allocation and execution constraints of the function can be dynamically adjusted to improve resource utilization and function execution efficiency.

[0035] Considering the possibility of exceptions and failures during function execution, this method also designs a fault-tolerant solution. By setting the number of retries and error handling rules for function execution, the fault tolerance of function execution can be improved to avoid the failure of the entire task due to exceptions in individual functions. At the same time, by utilizing the distributed characteristics of the serverless platform and the parallel execution path of the directed acyclic graph, a parallel processing framework is created to allow multiple non-dependent functions to be executed simultaneously, giving full play to the advantages of distributed computing and improving the concurrency and processing speed of data merging.

[0036] Finally, based on the parallel processing framework, function optimization configuration and fault tolerance scheme, the functions are executed in the order of priority in the task scheduling plan. After the functions corresponding to all nodes of the directed acyclic graph are executed, the final merge function is triggered. The final merge function is responsible for reading all intermediate results, integrating and processing them, and generating the final merged backup data. Through the execution of the final merge function, the finishing work of the entire data merging task is completed, ensuring the correctness and integrity of the merged results.

[0037] In this embodiment, by splitting the data merging task into independent functions, the automatic scaling and on-demand execution characteristics of the serverless computing platform can be used to dynamically adjust computing resources to adapt to data merging requirements of different scales and loads. By constructing a directed acyclic graph and task scheduling plan for function execution, the function trigger configuration is automatically generated, which reduces the workload of manual orchestration and scheduling and improves the efficiency and accuracy of task management. By collecting performance indicators of function execution, generating performance reports, and creating function optimization configurations, the resource allocation and execution constraints of the function can be dynamically adjusted to improve resource utilization and function execution efficiency. By setting the number of retries and error handling rules for function execution and building a fault-tolerant solution, the stability and reliability of task execution can be improved, and task failures caused by function exceptions can be reduced. By creating a parallel processing framework, the distributed characteristics of the serverless platform and the parallel execution path of the directed acyclic graph can be used to achieve concurrent execution of functions and improve the processing speed and throughput of data merging. Through the execution of the final merge function, all intermediate results are integrated and processed to generate the final merged backup data to ensure the consistency and availability of the merged results.

[0038] S102. Based on the characteristics and access patterns of the merged backup data, an adaptive optimization algorithm is used to automatically select a compression algorithm combination, and the merged backup data is hierarchically compressed to obtain compressed backup data. The compressed backup data is stored in a distributed storage system, and data block distribution is optimized. Life cycle prediction is performed on the compressed backup data stored in the distributed storage system to achieve hot and cold stratification of data. A multidimensional index structure is constructed at the same time, and the data dependency graph, classification results, and hot and cold stratification information are integrated into the multidimensional index.

[0039] Among them, by analyzing the merged backup data, the key features of the data, such as data type, data distribution, data correlation, etc., are extracted. These features will provide an important basis for the subsequent selection of compression algorithm combinations. Data exploration and statistical analysis methods can be used, such as calculating statistical indicators of data (mean, variance, skewness, etc.), drawing data distribution diagrams, analyzing data correlation matrices, etc. Collect and analyze historical access records of backup data, and mine data access patterns and rules, such as access frequency, access time distribution, access data volume, etc. The results of access pattern analysis will guide the selection of compression algorithms to optimize data compression and access performance. Data mining and machine learning methods can be used, such as association rule mining, sequence pattern mining, cluster analysis, etc.

[0040] Considering the characteristics and access patterns of backup data, an adaptive optimization algorithm is designed and implemented to automatically select the optimal compression algorithm combination. The algorithm needs to strike a balance between compression rate, compression speed, and data access performance. Heuristic optimization and evolutionary computing methods such as genetic algorithms, particle swarm optimization, and simulated annealing can be used.

[0041] During the data life cycle, the characteristics and access patterns of backup data may change. To adapt to these changes, a dynamic adjustment mechanism needs to be designed to monitor the changes in data characteristics and access patterns in real time, and automatically adjust the compression algorithm combination according to the changes to maintain the optimal compression performance. Adaptive control, reinforcement learning and other methods can be used to achieve dynamic optimization of the compression algorithm combination.

[0042] According to factors such as data importance and access frequency, the merged backup data is divided into multiple levels, such as key data, hot data, and ordinary data. Different compression strategies are used for data of different levels to balance the compression rate and access performance. Data classification and clustering methods can be used, such as K-means clustering, decision tree, support vector machine, etc.

[0043] For backup data of different levels, an adaptive optimization algorithm is used to select the best compression algorithm combination to achieve hierarchical compression. Key data uses an algorithm combination with a high compression rate and a low compression speed, hot data uses an algorithm combination with a low compression rate and a high compression speed, and ordinary data uses a compromise algorithm combination. Through hierarchical compression, the overall compression rate is maximized while ensuring the access performance of key data and hot data.

[0044] To ensure the integrity of compressed data, an integrity verification mechanism needs to be introduced during the compression process, such as calculating the hash value of the data before and after compression, and storing the hash value together with the compressed data. When the data is decompressed, the hash value is recalculated and compared with the stored hash value to verify the data integrity. Secure hash algorithms such as SHA-256 and MD5 can be used.

[0045] The compressed backup data is stored in a distributed storage system, such as Hadoop HDFS, Ceph, etc. Distributed storage systems can provide data storage services with high reliability, high scalability, and high throughput. The reliability and availability of data are guaranteed through mechanisms such as data sharding and replica replication.

[0046] According to the data access pattern and node performance, optimize the distribution of data blocks on distributed storage nodes to improve the locality and parallelism of data access. For example, store hot data blocks on high-performance nodes and store related data blocks on the same node or adjacent nodes. Data placement algorithms such as greedy algorithms and genetic algorithms can be used.

[0047] Perform lifecycle prediction on compressed backup data stored in distributed storage systems to identify cold data and hot data. Migrate cold data to low-cost, high-capacity storage devices (such as tape libraries) and keep hot data on high-performance storage devices (such as SSDs) to achieve hot and cold data stratification and reduce storage costs. Data lifecycle management strategies and data migration technologies can be used. Build a multidimensional index structure to integrate data dependency graphs, classification results, and hot and cold stratification information into the index. Multidimensional indexes can quickly locate and retrieve data and support complex data query and analysis operations. Commonly used multidimensional index structures include R-trees, KD-trees, octrees, etc.

[0048] In an optional embodiment, The life cycle of compressed backup data stored in the distributed storage system is predicted to achieve hot and cold data stratification, and a multi-dimensional index structure is constructed, including: Perform multi-dimensional life cycle analysis on the compressed backup data to obtain the historical access frequency and time distribution information of the data blocks, and build a data life cycle prediction model based on the historical access frequency and time distribution information; Using the data lifecycle prediction model, predicting the future access frequency and time distribution of each data block, setting a multi-level hot and cold data judgment threshold according to the predicted future access frequency and time distribution, and classifying each data block into a hot data block, a warm data block, and a cold data block based on the multi-level hot and cold data judgment threshold; Construct a data dependency graph to record the associations between the data blocks, allocate hot data blocks to high-performance storage devices, allocate warm data blocks to medium-performance storage devices, and allocate cold data blocks to low-cost storage devices. According to the classification results and storage locations of each data block, formulate a hierarchical storage optimization strategy including data pre-fetching, cache management, and data migration; Execute the tiered storage optimization strategy, record the access frequency, access delay and migration times of each data block, generate a performance optimization report, and determine the index dimensions and weights of the multidimensional index based on the performance optimization report; From the compressed backup data, metadata information is extracted according to the index dimensions and weights to generate an initial metadata set, which is preprocessed to obtain structured metadata. Based on the data distribution characteristics and query patterns of the structured metadata, performance indicators of different multidimensional index structures are calculated. Based on the performance indicators, the optimal multidimensional index structure is selected as the final multidimensional index structure.

[0049] For example, first, a multi-dimensional life cycle analysis is performed on the compressed backup data. By collecting and analyzing the historical access frequency and time distribution information of the data blocks, the access pattern and heat change law of the data blocks can be understood. Based on this historical information, a data life cycle prediction model is constructed to predict the future access frequency and time distribution of the data blocks. The data life cycle prediction model can use machine learning algorithms, such as time series prediction, regression analysis, etc., to learn the laws of data access patterns by training historical data, and predict future access conditions.

[0050] According to the prediction results of the data lifecycle prediction model, set multi-level hot and cold data judgment thresholds. The multi-level hot and cold data judgment thresholds are determined based on the predicted access frequency and time distribution of the data blocks, and are used to divide the data blocks into different heat levels. For example, three thresholds can be set to divide the data blocks with a predicted access frequency higher than the first threshold into hot data blocks, the data blocks with a predicted access frequency between the first threshold and the second threshold into warm data blocks, and the data blocks with a predicted access frequency lower than the second threshold into cold data blocks. By setting the thresholds reasonably, data can be managed in a hierarchical manner according to the access characteristics and business needs of the data.

[0051] After determining the hot and cold levels of data blocks, a data dependency graph is constructed to record the associations between data blocks. The data dependency graph reflects the logical relationships and access dependencies between data blocks, which can help optimize the storage layout and access paths of data. By analyzing the data dependency graph, data blocks with high correlation and frequent access together can be stored in close locations, reducing data access delays and overhead.

[0052] Data blocks are allocated to storage devices with different performances according to their hot and cold levels and data dependencies. Hot data blocks are allocated to high-performance storage devices, such as solid-state drives (SSDs) or memory, because they are frequently accessed and require fast responses. Warm data blocks are accessed relatively infrequently and can be allocated to medium-performance storage devices, such as high-speed hard disks (HDDs). Cold data blocks are accessed the least frequently and do not require high access speeds, so they can be allocated to low-cost storage devices, such as tape libraries or object storage. By allocating storage resources reasonably, you can reduce storage costs while ensuring data access performance.

[0053] In order to further optimize data access efficiency, a tiered storage optimization strategy including data pre-fetching, cache management and data migration is formulated. Data pre-fetching is to load the data that may be accessed to the higher-performance storage layer in advance according to the data access pattern and association relationship, thereby reducing the latency of data access. Cache management is to save frequently accessed data in the cache by reasonably setting the cache size, cache replacement strategy, etc., to accelerate data reading. Data migration is to dynamically adjust the distribution of data between different storage layers according to the data access frequency and storage cost, and migrate the data that is no longer frequently accessed to the storage layer with lower cost, saving storage space and cost.

[0054] In the process of executing the tiered storage optimization strategy, the performance indicators such as the access frequency, access delay, and migration times of each data block are recorded to generate a performance optimization report. The performance optimization report reflects the effect and improvement space of the tiered storage optimization strategy, and provides a basis for further optimization. According to the performance optimization report, the index dimensions and weights of the multidimensional index are determined. Multidimensional indexing is an efficient data retrieval technology that accelerates data query and filtering by indexing multiple attributes or dimensions of data. The selection of index dimensions and the setting of weights will affect the efficiency and space occupancy of the index, and need to be optimized according to the characteristics of the data and the query mode.

[0055] Extract metadata information from the compressed backup data based on index dimensions and weights to generate an initial metadata set. Metadata is information that describes data attributes and features, such as data identifiers, timestamps, keywords, etc. By extracting metadata, you can quickly locate and filter out data blocks that meet the query conditions, improving the efficiency of data retrieval. Preprocess the initial metadata set, such as cleaning, formatting, and normalization, to obtain structured metadata, which is convenient for subsequent index construction and query optimization.

[0056] Based on the data distribution characteristics and query patterns of structured metadata, the performance indicators of different multidimensional index structures are calculated. There are many types of multidimensional index structures, such as R-tree, KD-tree, octree, etc., which differ in space division, data organization and query algorithm. By analyzing the distribution characteristics and query patterns of data, such as data dimension, sparsity, query selectivity, etc., the applicability and performance of different index structures can be evaluated. Common performance indicators include index construction time, query response time, space occupancy, etc. According to the comparison results of performance indicators, the optimal multidimensional index structure is selected as the final indexing scheme.

[0057] In an optional embodiment, Based on the data distribution characteristics and query patterns of the structured metadata, the formula for calculating the performance indicators of different multidimensional index structures is as follows: ; Among them, M represents the multidimensional index performance index, P represents the number of different multidimensional indexes, and R j A represents the query response efficiency of the j-th multidimensional index. j represents the storage space occupied by the j-th multidimensional index, S j represents the number of dimensions of the jth index, γ represents the weight coefficient, δ represents the attenuation factor, and D j represents the update frequency of the jth index, Q represents the number of different queries, and T k Indicates the processing time of the kth query.

[0058] In this embodiment, by grading and storing data in layers, data blocks of different temperatures are allocated to suitable storage devices, making full use of storage resources with different performance and costs to avoid resource waste. By migrating cold data to low-cost storage devices, the occupation of high-performance storage devices is reduced, and the overall cost of the storage system is significantly reduced. Through optimization strategies such as data pre-fetching and cache management, frequently accessed data blocks are cached in the high-performance storage layer to reduce the delay of data access and improve the speed of data reading. Through data life cycle prediction and dynamic data migration, the distribution of data between different storage layers is dynamically adjusted according to the access mode and business needs of the data, so as to achieve the adaptability and flexibility of storage management. By constructing a multidimensional index, multiple attributes or dimensions of the data are indexed, the query and filtering operations of the data are accelerated, and the efficiency of data retrieval is significantly improved. By selecting a suitable multidimensional index structure and utilizing the spatial division and data organization characteristics of the index, complex query requirements such as multiple conditions and range queries are efficiently supported.

[0059] S103. When it is necessary to restore the relational database to a specified time point, the target backup data is quickly located using a multidimensional index structure, and the target backup data is decompressed using a parallel decompression technology corresponding to the compression algorithm combination to obtain the restored backup data, and the restored backup data is intelligently verified using data integrity constraint rules; if the intelligent verification passes, the restored backup data is imported into the relational database to complete the data recovery at the specified time point; if the intelligent verification fails, intelligent data repair and completion are performed, and then verification, verification and import are performed again until the intelligent verification passes.

[0060] First, when backing up data, in addition to storing the data itself, it is also necessary to record metadata information related to time, such as backup timestamp, transaction log sequence number, etc. These metadata information and backup data are combined to build a multidimensional index structure, such as R-tree, KD-tree, etc. The dimensions of the index can include timestamp, database table name, primary key, etc. The multidimensional index can quickly locate the target backup data at a specified time point.

[0061] When data recovery at a specified time point is required, firstly, according to the recovery time point, the backup data closest to the time point is retrieved in the multidimensional index. Index retrieval can significantly reduce the time overhead of data positioning. After the target backup data is retrieved, it needs to be decompressed. Since the backup data is usually compressed using a compression algorithm to save storage space, it needs to be decompressed during recovery. The use of parallel decompression technology corresponding to the compression algorithm can make full use of computing resources and accelerate the decompression process. For example, for block-based compression algorithms (such as LZ4, Snappy, etc.), the backup data can be divided into multiple independent blocks and decompressed in parallel to improve the decompression throughput.

[0062] The backup data obtained after decompression needs to be intelligently verified to ensure the integrity and consistency of the data. The restored backup data is verified using pre-defined data integrity constraint rules. These constraint rules may include primary key constraints, foreign key constraints, unique constraints, etc. By checking whether the data meets these constraints, errors and inconsistencies in the data can be found. The intelligent verification process can be implemented through SQL queries or specialized data verification tools.

[0063] In an optional embodiment, If the intelligent verification fails, intelligent data repair and completion will be performed, including: If the intelligent verification fails, intelligent data repair is performed, and the errors in the data are automatically corrected using the integrity constraint rules to obtain preliminary repaired data. The integrity constraint rules are converted into machine-executable repair operations, and the constraint satisfaction problem solving algorithm is used to infer a repair solution that satisfies the constraint conditions. According to the repair solution, the search direction of the repair algorithm is guided to further optimize the preliminary repaired data to obtain the target repaired data; Apply the association rule mining algorithm to the target repair data to obtain the dependency relationship between attributes. According to the dependency relationship between attributes and the values ​​of known attributes, the values ​​of missing attributes are inferred to obtain preliminary completed data. The initially completed data is represented as a high-dimensional sparse matrix, a matrix decomposition algorithm is used to obtain potential patterns and similarities in the data, and a factor matrix is ​​obtained. The factor matrix is ​​used to calculate missing values, and the initially completed data is secondary completed according to the missing values. A collaborative filtering algorithm is used to predict the missing numerical types in the secondary completed data according to historical data of similar users or projects, and the collaborative filtering completed data is obtained. According to the collaborative filtering completion data obtained, a deep learning-based generative model is trained, and the trained generative model is used to generate missing attribute values ​​to obtain the generative model completion data. The completed data is represented as a graph structure, and graph neural network modeling and learning are used. Through message passing and aggregation operations, the existence probability of missing node attributes or edges is inferred, and finally the completed graph structure data is obtained to achieve intelligent data repair and completion.

[0064] Exemplarily, first, when the data fails to pass the intelligent verification, intelligent data repair is performed. Predefined integrity constraint rules are used to automatically detect and correct errors in the data. Integrity constraint rules refer to conditions or restrictions that data must meet, such as value range, format specifications, uniqueness, etc. By converting integrity constraint rules into machine-executable repair operations, constraint satisfaction problem solving algorithms, such as constraint programming, SAT solvers, etc., are used to infer repair solutions that meet the constraints. Constraint satisfaction problems refer to finding a set of variable values ​​that satisfies all constraints at the same time under given constraints. According to the inferred repair solution, the search direction of the repair algorithm is guided, the preliminary repaired data is optimized, and the target repair data that meets the integrity constraints is obtained.

[0065] Next, we apply the association rule mining algorithm to the target repair data to find the dependencies between attributes in the data. Association rule mining is a data mining technique used to find interesting association patterns between data items. By analyzing the dependencies between attributes, we can infer the values ​​of missing attributes based on the values ​​of known attributes. For example, if there is an association rule "occupation = teacher -> education = bachelor degree and above", then for data with occupation = teacher but missing education attribute, we can infer that the education is bachelor degree and above based on this rule. We use association rules to infer and fill in the missing attributes to obtain preliminary completed data.

[0066] In order to further improve the accuracy of data completion, the initially completed data is represented as a high-dimensional sparse matrix. A high-dimensional sparse matrix refers to a matrix with high dimensions and most elements are zero. Matrix decomposition algorithms, such as singular value decomposition (SVD) and non-negative matrix factorization (NMF), are used to reduce the dimension and extract features of the high-dimensional sparse matrix, obtain the potential patterns and similarities in the data, and obtain the factor matrix. The factor matrix refers to a low-dimensional dense matrix obtained by matrix decomposition, which captures the main features and structural information of the original matrix. Using the factor matrix, the missing values ​​are calculated and the initially completed data is secondary completed.

[0067] In order to make full use of the historical data of similar users or projects, the collaborative filtering algorithm is used to predict the missing numerical attributes in the secondary completion data. Collaborative filtering is a recommendation algorithm based on user or project similarity. It predicts the target user's preference or rating for a project by analyzing the similarity between users or projects. The users or projects in the data are mapped to a low-dimensional space, the similarity between them is calculated, and the missing numerical attributes of the target user or project are predicted based on the historical data of similar users or projects to obtain the data completed by collaborative filtering.

[0068] After obtaining the data for collaborative filtering completion, deep learning technology is further used to generate the values ​​of missing attributes. Generative models based on deep learning, such as generative adversarial networks (GANs) and variational autoencoders (VAEs), are trained to learn the intrinsic distribution and feature representation of data. Generative models are a type of model that generates new data similar to training data by learning data distribution. Using the trained generative model, the values ​​of missing attributes are generated based on known attributes to obtain data completed by the generative model.

[0069] Finally, the completed data is represented as a graph structure and modeled and learned using a graph neural network. A graph neural network is a deep learning model suitable for graph structured data. It learns the feature representation of nodes and the relationship between nodes through message passing and aggregation operations. The entities in the data are represented as nodes, and the relationship between entities is represented as edges to construct a data graph. The graph neural network is used to encode and reason about the data graph. Through message passing and aggregation operations, the probability of the existence of missing node attributes or edges is inferred, and the completed graph structured data is finally obtained.

[0070] In this embodiment, by utilizing integrity constraint rules, errors in the data are automatically detected and corrected to improve the accuracy and consistency of the data. By combining association rule mining, matrix decomposition, collaborative filtering, and deep learning technologies, missing data is intelligently supplemented from multiple dimensions to improve the integrity of the data. Through a progressive repair and completion strategy, the integrity, accuracy, and consistency of the data are gradually improved, and the data quality is comprehensively improved. Through association rule mining and deep learning, the hidden patterns, associations, and feature representations in the data are discovered to mine the potential value of the data. The graph neural network is used to model and learn the supplemented data to improve the generalization ability of the model for unknown data and enhance the availability of the data. Through intelligent data repair and completion methods, the workload of manually processing data errors and missing data is reduced, and the efficiency of data processing is improved.

[0071] In an optional embodiment, The method further comprises: Use causal discovery algorithms to learn the causal relationship between variables from data, build a causal graph model, determine the parent node and child node of the missing value node based on the constructed causal graph, and establish the dependency relationship between the missing value and the causally related variables; Using the conditional probability distribution model, the conditional probability distribution of the missing value node is calculated when the parent node value is given, and the preliminary repair and completion results of the missing value are generated according to the estimated conditional probability distribution; Construct corresponding simulation scenarios for the preliminary repair and completion results. In the simulation scenarios, design adaptive model selection and parameter tuning mechanisms, automatically select causal discovery algorithms and causal effect estimation methods according to data characteristics and task requirements, and use causal graph models and the selected causal effect estimation methods to estimate the causal effects of the preliminary repair and completion results on key variables; According to the estimated results of the causal effect, the repair and completion schemes are optimized, and the scheme with the best causal effect is selected as the final repair and completion result.

[0072] Exemplarily, first, a causal discovery algorithm is used to learn the causal relationship between variables from the data and construct a causal graph model. Causal discovery refers to inferring the causal relationship between variables from the observed data, that is, identifying the causal direction and causal strength. Commonly used causal discovery algorithms include the PC algorithm based on conditional independence test, the GES algorithm based on score search, and the LiNGAM algorithm based on the functional causal model. These algorithms infer the causal direction and causal strength between variables by analyzing the conditional independence, information theory measurement or non-Gaussian hypothesis between variables, and construct a causal graph model. The causal graph model is a directed acyclic graph, where nodes represent variables, directed edges represent causal relationships, the direction of the edge represents the causal direction, and the weight of the edge represents the causal strength.

[0073] Based on the constructed causal graph, the parent node and child node of the missing value node are determined, and the dependency relationship between the missing value and the causal-related variables is established. In the causal graph, if a node is the parent node of another node, it means that the former is the direct cause of the latter. If a node is the child node of another node, it means that the former is the direct result of the latter. By analyzing the position and connection relationship of the missing value node in the causal graph, its parent node and child node, that is, which variables the missing value node has a direct causal dependency relationship with, can be determined.

[0074] The conditional probability distribution model is used to calculate the conditional probability distribution of the missing value node given the value of the parent node. The conditional probability distribution describes the probability distribution of the target variable given the value of certain variables. For the missing value node, the conditional probability distribution of the missing value node is estimated based on the value of its parent node. Common conditional probability distribution estimation methods include maximum likelihood estimation, Bayesian estimation, kernel density estimation, etc. Based on the estimated conditional probability distribution, possible values ​​of the missing value node are generated as the preliminary repair and completion results.

[0075] Construct corresponding simulation scenarios for the preliminary repair and completion results, and design adaptive model selection and parameter tuning mechanisms. The simulation scenario refers to the construction of a virtual environment similar to the real scenario based on the preliminary repair and completion results, which is used to evaluate the effects of repair and completion. The adaptive model selection and parameter tuning mechanism refers to the automatic selection of appropriate causal discovery algorithms and causal effect estimation methods based on data characteristics and task requirements, and the adjustment of the algorithm's hyperparameters to optimize model performance. Commonly used model selection methods include cross-validation, information criterion, Bayesian model selection, etc. Parameter tuning methods include grid search, random search, Bayesian optimization, etc.

[0076] In the simulation scenario, the causal graph model and the selected causal effect estimation method are used to estimate the causal effect of the initial repair and completion results on the key variables. The causal effect refers to the degree of influence of the change of one variable on another variable, reflecting the strength of the causal relationship between the variables. Commonly used causal effect estimation methods include propensity score matching, causal tree, causal forest, etc. By estimating the causal effect of the initial repair and completion results on the key variables, the effectiveness and impact of the repair and completion are evaluated.

[0077] Optimize the repair and completion schemes based on the estimated results of the causal effect. Compare the causal effects of different repair and completion schemes on key variables, and select the scheme with the best causal effect as the final repair and completion result. The best causal effect means that after repair and completion, the causal effect of the key variable is closest to the actual situation, or the impact on downstream tasks is the most positive. By optimizing the repair and completion schemes, the quality and reliability of data repair and completion can be improved.

[0078] In this embodiment, the causal relationship between variables is automatically learned from the data through the causal discovery algorithm, and a causal graph model is constructed to reveal the inherent causal structure and dependency of the data. The conditional probability distribution model is used to estimate the conditional probability distribution of the missing value based on the parent node information of the missing value node, and generate reasonable repair and completion results. An adaptive model selection and parameter tuning mechanism is designed to automatically select the optimal causal discovery algorithm and causal effect estimation method according to the data characteristics and task requirements, and adjust the hyperparameters to improve the model performance. The causal effect of the repair and completion results on the key variables is evaluated by using the causal graph model and the causal effect estimation method, and the effectiveness and impact of the repair and completion are quantified. According to the estimated results of the causal effect, the repair and completion scheme is optimized, the scheme with the best causal effect is selected, and the quality and reliability of data repair and completion are improved. Through causal discovery and causal effect estimation, missing data is automatically repaired and completed, the integrity, accuracy and consistency of the data are improved, and the foundation for subsequent data analysis and mining is laid.

[0079] In an optional embodiment, first, according to the data recovery requirements, the scope of data restoration is confirmed, and the data is imported into a temporary table. Then, it is determined whether the database table to be restored adopts the sub-library and sub-table strategy. If so, it is necessary to determine the sub-library and sub-table algorithm rules, and clean the data in the temporary table, map it to the sub-library and sub-table fields of the target table, and obtain the temporary table corresponding to the target table and its sub-library and sub-table implementation algorithm. Next, the data restoration task is configured, which mainly includes the following contents: target data source configuration, target library table information, and table information to be restored. Among them, the target data source configuration includes information such as the host address, port number, user name and password of the target data source, which is used to establish a connection with the target database. The target library table information includes the target database name, the target data table name, and the sub-library and sub-table configuration information, which is used to determine the target location and strategy for data restoration. The table information to be restored includes relevant information of the data warehouse source table, which is used to identify the source of data to be restored. By configuring this information, a mapping relationship between the temporary table and the target table is established to prepare for data restoration.

[0080] When actually performing data restoration tasks, you also need to consider the control of data transmission rate. By determining whether current limiting is required, dynamically adjusting the number of data synchronization threads and the size of data synchronization batches according to the performance of the current target database table, the data transmission rate can be controlled. This can avoid excessive performance pressure on the target database table and ensure the stability and reliability of the data restoration process.

[0081] After the data restoration task is started, the system will gradually restore the data in the temporary table to the target database table according to the configured mapping relationship. During the restoration process, the system will record the number of synchronized data in real time and monitor the status of data synchronization. At the same time, in order to prevent abnormalities in the data restoration process that lead to data inconsistency, the system will also prepare for data rollback to ensure data integrity and consistency.

[0082] In this embodiment, by importing data into a temporary table and using a library-splitting algorithm to clean and map the data, the data can be efficiently restored from the temporary table to the target library table, thereby improving the efficiency of data recovery. The method can adapt to different library-splitting strategies, and by configuring the library-splitting algorithm rules, the data can be restored to the target library table that adopts the library-splitting, thereby improving the flexibility of data recovery. By judging whether current limiting is required and dynamically adjusting the number of data synchronization threads and batch size according to the performance of the target library table, the data transmission rate can be effectively controlled to avoid excessive performance pressure on the target library table, thereby ensuring the stability and reliability of the data recovery process. During the data restoration process, by recording the number of synchronized data in real time and monitoring the synchronization status, and making preparations for data rollback, the integrity and consistency of the data restoration process can be ensured, thereby avoiding the problem of data inconsistency. By configuring the target data source, the target library table information, and the table to be restored information, a mapping relationship between the temporary table and the target table is established, thereby making the configuration and management of the data recovery task simpler and more intuitive, thereby improving the ease of use and maintainability of the system.

[0083] Figure 2 FIG. 1 is a schematic diagram of a structure of a relational database backup and recovery system based on big data according to an embodiment of the present invention. Figure 2 As shown, the system comprises: The first unit is used to obtain full backup data and incremental backup data of a relational database, intelligently classify the full backup data and incremental backup data using a machine learning algorithm, and build a data dependency graph based on the classification results, analyze the correlation and integrity constraints between data, use a graph neural network to analyze the data dependency graph, identify changes in data structure, and automatically generate a data merging strategy based on the identification results, adopt a parallel computing framework, and intelligently merge the full backup data and incremental backup data according to the data merging strategy to obtain merged backup data; The second unit is used to automatically select a compression algorithm combination based on the characteristics and access mode of the merged backup data using an adaptive optimization algorithm, perform hierarchical compression on the merged backup data to obtain compressed backup data, store the compressed backup data in a distributed storage system, optimize data block distribution, perform life cycle prediction on the compressed backup data stored in the distributed storage system, realize hot and cold stratification of data, and construct a multidimensional index structure to integrate the data dependency graph, classification results, and hot and cold stratification information into the multidimensional index; The third unit is used to use a multidimensional index structure to quickly locate target backup data when it is necessary to restore the relational database to a specified time point, and to decompress the target backup data using a parallel decompression technology corresponding to the compression algorithm combination to obtain restored backup data, and to perform intelligent verification on the restored backup data using data integrity constraint rules; if the intelligent verification passes, the restored backup data is imported into the relational database to complete data recovery at the specified time point; if the intelligent verification fails, intelligent data repair and completion are performed, and then verification, verification and import are performed again until the intelligent verification passes.

[0084] Figure 3 It is a framework diagram of the scheme of the present invention, such as Figure 3 As shown, the entire process is divided into three parts: data backup module, merging and compression, and big data platform.

[0085] Among them, the data backup module includes a full backup module and an incremental backup module, which are responsible for obtaining the full data and incremental data of the relational database respectively.

[0086] Merge and compression: The acquired full backup data and incremental backup data will be transmitted to this module for intelligent merging processing, and then the merged data will be compressed using a hierarchical compression algorithm.

[0087] Big data platform: The compressed backup data will be stored in the distributed file system and a multi-dimensional index will be constructed. When data recovery is needed, the platform will quickly locate the target data, decompress the data using parallel decompression technology, and verify, repair and import the data through the computing engine, and finally complete the data recovery at the specified time point.

[0088] The system integrates big data technologies such as distributed storage and parallel computing, which can significantly improve the efficiency and reliability of relational database backup and recovery and realize the management of massive data.

[0089] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0090] According to a fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0091] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A relational database backup and recovery method based on big data, characterized in that: include: Obtain the full backup data and incremental backup data of the relational database, use the machine learning algorithm to intelligently classify the full backup data and incremental backup data, and build a data dependency graph based on the classification results, analyze the correlation and integrity constraints between the data, use the graph neural network to analyze the data dependency graph, identify changes in the data structure, and automatically generate a data merging strategy based on the identification results. Use a parallel computing framework to intelligently merge the full backup data and incremental backup data according to the data merging strategy to obtain the merged backup data; Based on the characteristics and access patterns of the merged backup data, a compression algorithm combination is automatically selected using an adaptive optimization algorithm, the merged backup data is hierarchically compressed to obtain compressed backup data, the compressed backup data is stored in a distributed storage system, and data block distribution is optimized, life cycle prediction is performed on the compressed backup data stored in the distributed storage system, cold and hot stratification of data is achieved, and a multidimensional index structure is constructed at the same time, and the data dependency graph, classification results, and cold and hot stratification information are integrated into the multidimensional index; When it is necessary to restore the relational database to a specified time point, the multidimensional index structure is used to quickly locate the target backup data, and the parallel decompression technology corresponding to the compression algorithm combination is used to decompress the target backup data to obtain the restored backup data, and the restored backup data is intelligently verified using data integrity constraint rules; if the intelligent verification passes, the restored backup data is imported into the relational database to complete the data recovery at the specified time point; if the intelligent verification fails, intelligent data repair and completion are performed, and then verification, verification and import are performed again until the intelligent verification passes.

2. The method according to claim 1, characterized in that Based on the classification results, a data dependency graph is constructed to analyze the correlation and integrity constraints between data. The data dependency graph is analyzed using a graph neural network to identify changes in data structure, and a data merging strategy is automatically generated based on the identification results. A parallel computing framework is used to intelligently merge the full backup data and incremental backup data according to the data merging strategy. The merged backup data includes: Based on the classification results, identify the primary key and unique key fields of each table, compare these fields one-to-one with all the fields of all other tables in the database, identify potential foreign key relationships by comparing the similarity of field names, data types, lengths, and value ranges, calculate the data overlap and distribution similarity between the identified potential related fields, and determine the strength of correlation between the fields; Based on the identified foreign key relationships and field correlation strengths, a data dependency graph is constructed, where nodes represent data tables and edges represent relationships between tables. The constructed data dependency graph is input into the graph neural network model, and through multi-layer graph convolution and pooling operations combined with the attention mechanism, an embedding vector for each data table node is generated; Calculate the cosine similarity between the embedding vector of the current timestamp node and the embedding vector of the historical timestamp node, judge the changes in the data table according to the preset similarity threshold, identify newly added tables, tables with data volume growth exceeding the preset threshold, and large-scale data update events, and generate data structure change recognition results; According to the results of data structure change identification, a data merging strategy is automatically generated. Based on the data merging strategy, the backup data is divided into independent processing units according to the data table, and a directed acyclic graph of the merging task is constructed. At the same time, the distributed computing framework is utilized, and the fault tolerance and retry mechanism are set up to perform parallel merging operations and generate the final merged backup data.

3. The method according to claim 2, characterized in that Construct a directed acyclic graph of the merge task, use the distributed computing framework, set up fault tolerance and retry mechanisms, perform parallel merge operations, and generate the final merged backup data including: In the distributed computing framework, according to the pre-set serverless computing model, the data merging task is split into multiple independent functions, where each function corresponds to a node in the directed acyclic graph. Each function is deployed to the serverless platform, and the function trigger configuration is generated according to the edge relationship in the directed acyclic graph. And the task scheduling plan is generated according to the priority of function execution; According to the task scheduling plan and function trigger configuration, the serverless platform is integrated with the message queue service to create a data transmission channel. The object storage service of the serverless platform is used to establish a result repository for storing intermediate results of function execution. The monitoring tool of the serverless platform is used to collect the number of function calls, execution time, memory usage and error rate, and generate a performance report. Based on the performance report and the directed acyclic graph structure, a function optimization configuration including memory allocation and timeout time is created; According to the function optimization configuration, the number of retries and error handling rules for function execution are set, a fault-tolerant solution is generated, and a parallel processing framework for simultaneously executing multiple non-dependency functions is created by utilizing the distributed characteristics of the serverless platform and the parallel execution path of the directed acyclic graph; Based on the parallel processing framework, function optimization configuration and fault-tolerant scheme, functions are executed in the order of priority in the task scheduling plan. After the functions corresponding to all nodes of the directed acyclic graph are executed, the final merging function is triggered. The final merging function reads all intermediate results, integrates them and generates the final merged backup data.

4. The method according to claim 1, characterized in that The life cycle of compressed backup data stored in the distributed storage system is predicted to achieve hot and cold data stratification, and a multi-dimensional index structure is constructed, including: Perform multi-dimensional life cycle analysis on the compressed backup data to obtain the historical access frequency and time distribution information of the data blocks, and build a data life cycle prediction model based on the historical access frequency and time distribution information; Using the data lifecycle prediction model, predicting the future access frequency and time distribution of each data block, setting a multi-level hot and cold data judgment threshold according to the predicted future access frequency and time distribution, and classifying each data block into a hot data block, a warm data block, and a cold data block based on the multi-level hot and cold data judgment threshold; Construct a data dependency graph to record the associations between the data blocks, allocate hot data blocks to high-performance storage devices, allocate warm data blocks to medium-performance storage devices, and allocate cold data blocks to low-cost storage devices. According to the classification results and storage locations of each data block, formulate a hierarchical storage optimization strategy including data pre-fetching, cache management, and data migration; Execute the tiered storage optimization strategy, record the access frequency, access delay and migration times of each data block, generate a performance optimization report, and determine the index dimensions and weights of the multidimensional index based on the performance optimization report; From the compressed backup data, metadata information is extracted according to the index dimensions and weights to generate an initial metadata set, which is preprocessed to obtain structured metadata. Based on the data distribution characteristics and query patterns of the structured metadata, performance indicators of different multidimensional index structures are calculated. Based on the performance indicators, the optimal multidimensional index structure is selected as the final multidimensional index structure.

5. The method according to claim 4, characterized in that Based on the data distribution characteristics and query patterns of the structured metadata, the formula for calculating the performance indicators of different multidimensional index structures is as follows: ； Among them, M represents the multidimensional index performance index, P represents the number of different multidimensional indexes, and R j A represents the query response efficiency of the j-th multidimensional index. j represents the storage space occupied by the j-th multidimensional index, S j represents the number of dimensions of the jth index, γ represents the weight coefficient, δ represents the attenuation factor, and D j represents the update frequency of the jth index, Q represents the number of different queries, and T k Indicates the processing time of the kth query.

6. The method according to claim 1, characterized in that If the intelligent verification fails, intelligent data repair and completion will be performed, including: If the intelligent verification fails, intelligent data repair is performed, and the errors in the data are automatically corrected using the integrity constraint rules to obtain preliminary repaired data. The integrity constraint rules are converted into machine-executable repair operations, and the constraint satisfaction problem solving algorithm is used to infer a repair solution that satisfies the constraint conditions. According to the repair solution, the search direction of the repair algorithm is guided to further optimize the preliminary repaired data to obtain the target repaired data; Apply the association rule mining algorithm to the target repair data to obtain the dependency relationship between attributes. According to the dependency relationship between attributes and the values ​​of known attributes, the values ​​of missing attributes are inferred to obtain preliminary completed data. The initially completed data is represented as a high-dimensional sparse matrix, a matrix decomposition algorithm is used to obtain potential patterns and similarities in the data, and a factor matrix is ​​obtained. The factor matrix is ​​used to calculate missing values, and the initially completed data is secondary completed according to the missing values. A collaborative filtering algorithm is used to predict the missing numerical types in the secondary completed data according to historical data of similar users or projects, and the collaborative filtering completed data is obtained. According to the collaborative filtering completion data obtained, a deep learning-based generative model is trained, and the trained generative model is used to generate missing attribute values ​​to obtain the generative model completion data. The completed data is represented as a graph structure, and graph neural network modeling and learning are used. Through message passing and aggregation operations, the existence probability of missing node attributes or edges is inferred, and finally the completed graph structure data is obtained to achieve intelligent data repair and completion.

7. The method according to claim 1, characterized in that The method further comprises: Use causal discovery algorithms to learn the causal relationship between variables from data, build a causal graph model, determine the parent node and child node of the missing value node based on the constructed causal graph, and establish the dependency relationship between the missing value and the causally related variables; Using the conditional probability distribution model, the conditional probability distribution of the missing value node is calculated when the parent node value is given, and the preliminary repair and completion results of the missing value are generated according to the estimated conditional probability distribution; Construct corresponding simulation scenarios for the preliminary repair and completion results. In the simulation scenarios, design adaptive model selection and parameter tuning mechanisms, automatically select causal discovery algorithms and causal effect estimation methods according to data characteristics and task requirements, and use causal graph models and the selected causal effect estimation methods to estimate the causal effects of the preliminary repair and completion results on key variables; According to the estimated results of the causal effect, the repair and completion schemes are optimized, and the scheme with the best causal effect is selected as the final repair and completion result.

8. A relational database backup and recovery system based on big data, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to obtain full backup data and incremental backup data of a relational database, intelligently classify the full backup data and incremental backup data using a machine learning algorithm, and build a data dependency graph based on the classification results, analyze the correlation and integrity constraints between data, use a graph neural network to analyze the data dependency graph, identify changes in data structure, and automatically generate a data merging strategy based on the identification results, adopt a parallel computing framework, and intelligently merge the full backup data and incremental backup data according to the data merging strategy to obtain merged backup data; The second unit is used to automatically select a compression algorithm combination based on the characteristics and access mode of the merged backup data using an adaptive optimization algorithm, perform hierarchical compression on the merged backup data to obtain compressed backup data, store the compressed backup data in a distributed storage system, optimize data block distribution, perform life cycle prediction on the compressed backup data stored in the distributed storage system, realize hot and cold stratification of data, and construct a multidimensional index structure to integrate the data dependency graph, classification results, and hot and cold stratification information into the multidimensional index; The third unit is used to use a multidimensional index structure to quickly locate target backup data when it is necessary to restore the relational database to a specified time point, and to decompress the target backup data using a parallel decompression technology corresponding to the compression algorithm combination to obtain restored backup data, and to perform intelligent verification on the restored backup data using data integrity constraint rules; if the intelligent verification passes, the restored backup data is imported into the relational database to complete data recovery at the specified time point; if the intelligent verification fails, intelligent data repair and completion are performed, and then verification, verification and import are performed again until the intelligent verification passes.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images