Big data-based relational database backup and recovery method and system

Abstract

The application provides a relational database backup recovery method and system based on big data, relates to the technical field of database backup recovery, and comprises the following steps: obtaining full backup data and incremental backup data of a relational database to obtain merged backup data; based on the features and access modes of the merged backup data, a multi-dimensional index structure is constructed, and data dependency graphs, classification results and hot and cold layering information are integrated into the multi-dimensional index; target backup data is decompressed to obtain recovered backup data, and the recovered backup data is intelligently checked by using data integrity constraint rules; if the intelligent checking is passed, the recovered backup data is imported into the relational database to complete data recovery at a specified time point; if the intelligent checking is not passed, intelligent data repair and completion are performed, and then verification, checking and import are performed again until the intelligent checking is passed.

Description

TECHNICAL FIELD

[0001] The present application relates to database backup and recovery technology, and particularly relates to a relational database backup and recovery method and system based on big data. BACKGROUND

[0002] With the continuous growth of data volume and the improvement of business system complexity, the backup and recovery of relational databases face many challenges. Traditional backup and recovery methods usually adopt a combination of full backup and incremental backup, but in the case of large backup data volume, complex data structure, strong data correlation, etc., the data merging, compression, storage and recovery process is time-consuming and inefficient, and it is difficult to meet the demand of fast recovery to a specified time point. In addition, the problems of high storage cost of backup data, large data redundancy, difficulty in guaranteeing data integrity, etc. also need to be solved.

[0003] In order to cope with the above challenges, it is urgent to develop a relational database backup and recovery method and system based on big data technology, to realize intelligent classification, automatic merging, fast positioning and intelligent verification of backup data, to improve the efficiency, reliability and flexibility of data backup and recovery, to reduce storage cost, and to guarantee the integrity and consistency of data. SUMMARY

[0004] The embodiment of the present application provides a relational database backup and recovery method and system based on big data, which can solve the problems in the prior art.

[0005] The first aspect of the embodiment of the present application is,

[0006] A relational database backup and recovery method based on big data is provided, comprising:

[0007] Full backup data and incremental backup data of a relational database are obtained, machine learning algorithms are used to intelligently classify the full backup data and the incremental backup data, and a data dependency graph is constructed based on the classification results to analyze the correlation and integrity constraints between data. A graph neural network is used to analyze the data dependency graph, identify changes in data structure, and automatically generate a data merging strategy according to the identification results. A parallel computing framework is used to intelligently merge and process the full backup data and the incremental backup data according to the data merging strategy, and the merged backup data is obtained.

[0008] Based on the features and access patterns of the merged backup data, an adaptive optimization algorithm is used to automatically select a compression algorithm combination, perform hierarchical compression on the merged backup data to obtain compressed backup data, store the compressed backup data in a distributed storage system, optimize data block distribution, predict the life cycle of the compressed backup data stored in the distributed storage system, realize cold and hot layering of data, and construct a multi-dimensional index structure, integrate data dependency graph, classification results and cold and hot layering information into the multi-dimensional index.

[0009] When it is necessary to restore the relational database to a specified time point, the multi-dimensional index structure is used to quickly locate the target backup data, parallel decompression technology corresponding to the compression algorithm combination is used to decompress the target backup data to obtain restored backup data, and the data integrity constraint rules are used to intelligently check the restored backup data; if the intelligent check passes, the restored backup data is imported into the relational database to complete the data restoration at the specified time point; if the intelligent check fails, intelligent data repair and completion are performed, and then verification, checking and import are performed again until the intelligent check passes.

[0010] In an optional embodiment,

[0011] Based on the classification results, a data dependency graph is constructed to analyze the correlation and integrity constraints between data, a graph neural network is used to analyze the data dependency graph to identify changes in data structure, and a data merging strategy is automatically generated according to the identification results. Using a parallel computing framework, the full backup data and the incremental backup data are intelligently merged according to the data merging strategy to obtain merged backup data, including:

[0012] Based on the classification results, the primary key and unique key fields of each table are identified, and these fields are compared one by one with all fields of all other tables in the database. By comparing the similarity of field names, data types, lengths and value ranges, potential foreign key relationships are identified, the data overlap and distribution similarity between the identified potential related fields are calculated, and the correlation strength between the fields is determined.

[0013] According to the identified foreign key relationships and field correlation strength, a data dependency graph is constructed, wherein the nodes represent data tables and the edges represent the relationships between the tables. The constructed data dependency graph is input into a graph neural network model, and through multi-layer graph convolution and pooling operations combined with an attention mechanism, an embedding vector of each data table node is generated.

[0014] The cosine similarity of the embedding vector of the current timestamp node and the embedding vector of the historical timestamp node is calculated, the change of the data table is judged according to a preset similarity threshold, a newly added table, a table with data growth exceeding a preset threshold and a large-scale data update event are recognized, and a data structure change recognition result is generated.

[0015] According to the data structure change recognition result, a data merging strategy is automatically generated, the backup data is divided into independent processing units according to the data table based on the data merging strategy, a directed acyclic graph of the merging task is constructed, a distributed computing framework is utilized, a fault tolerance and retry mechanism is set, a parallel merging operation is executed, and final merged backup data is generated.

[0016] In an optional embodiment,

[0017] The directed acyclic graph of the merging task is constructed, a distributed computing framework is utilized, a fault tolerance and retry mechanism is set, a parallel merging operation is executed, and final merged backup data is generated, including:

[0018] In the distributed computing framework, the data merging task is split into multiple independent functions according to a pre-set serverless computing model, each function corresponds to a node in the directed acyclic graph, each function is deployed to a serverless platform, function trigger configurations are generated according to the edge relationship in the directed acyclic graph, and a task scheduling plan is generated according to the priority of function execution;

[0019] According to the task scheduling plan and the function trigger configuration, the serverless platform and the message queue service are integrated to create a data transmission channel, the object storage service of the serverless platform is used to establish a result storage library for storing intermediate results of function execution, the monitoring tool of the serverless platform is used to collect function call times, execution time, memory usage and error rate to generate a performance report, and based on the performance report and the directed acyclic graph structure, a function optimization configuration including memory allocation and timeout is created;

[0020] According to the function optimization configuration, the number of retries and error handling rules of function execution are set to generate a fault tolerance scheme, and the distributed characteristics of the serverless platform and the parallel execution path of the directed acyclic graph are used to create a parallel processing framework for simultaneously executing multiple independent functions;

[0021] Based on the parallel processing framework, the function optimization configuration and the fault tolerance scheme, the functions are executed in the order of priority in the task scheduling plan, and after the functions corresponding to all nodes in the directed acyclic graph are executed, a final merging function is triggered, which reads all intermediate results, integrates and processes them to generate final merged backup data.

[0022] In an optional embodiment,

[0023] The compressed backup data stored in the distributed storage system is subjected to life cycle prediction, realizing cold and hot stratification of data, and a multi-dimensional index structure is constructed, including:

[0024] The compressed backup data is subjected to multi-dimensional life cycle analysis, the historical access frequency and time distribution information of the data block are acquired, and a data life cycle prediction model is constructed based on the historical access frequency and time distribution information;

[0025] The data life cycle prediction model is used to predict the future access frequency and time distribution of each data block, multi-level cold and hot data judgment thresholds are set according to the predicted future access frequency and time distribution, and each data block is classified as a hot data block, a warm data block and a cold data block based on the multi-level cold and hot data judgment thresholds;

[0026] A data dependency graph is constructed to record the association relationship between the data blocks, the hot data blocks are allocated to high-performance storage devices, the warm data blocks are allocated to medium-performance storage devices, and the cold data blocks are allocated to low-cost storage devices, and a stratified storage optimization strategy including data prefetching, cache management and data migration is formulated according to the classification results and storage locations of the data blocks;

[0027] The stratified storage optimization strategy is executed, the access frequency, access delay and migration times of each data block are recorded, a performance optimization report is generated, and the index dimensions and weights of the multi-dimensional index are determined based on the performance optimization report;

[0028] From the compressed backup data, metadata information is extracted according to the index dimensions and weights to generate an initial metadata set, the initial metadata set is preprocessed to obtain structured metadata, the performance indicators of different multi-dimensional index structures are calculated based on the data distribution characteristics and query patterns of the structured metadata, and the optimal multi-dimensional index structure is selected as the final multi-dimensional index structure according to the performance indicators.

[0029] In an optional embodiment,

[0030] The formula for calculating the performance indicators of different multi-dimensional index structures based on the data distribution characteristics and query patterns of the structured metadata is as follows:

[0031] ;

[0032] Wherein, M represents the multi-dimensional index performance indicator, P represents the number of different multi-dimensional indexes, R j represents the query response efficiency of the jth multi-dimensional index, A j represents the storage space occupation of the jth multi-dimensional index, S j represents the dimension number of the jth index, γ represents the weight coefficient, δ represents the attenuation factor, D jrepresents the update frequency of the jth index, Q represents the number of different queries, T k represents the processing time of the kth query.

[0033] In an optional embodiment,

[0034] If the intelligent verification fails, intelligent data repair and completion includes:

[0035] If the intelligent verification fails, intelligent data repair is performed, errors in the data are automatically corrected using integrity constraint rules to obtain preliminary repaired data, the integrity constraint rules are converted into machine executable repair operations, a constraint satisfaction problem solving algorithm is used to infer a repair scheme that satisfies the constraint conditions, the repair scheme is used to guide the search direction of the repair algorithm, and the preliminary repaired data is further optimized to obtain target repaired data;

[0036] The correlation rule mining algorithm is applied to the target repaired data to obtain the dependency relationship between attributes, and the values of the missing attributes are inferred according to the dependency relationship between the attributes and the values of the known attributes to obtain preliminary completed data;

[0037] The preliminary completed data is represented as a high-dimensional sparse matrix, a matrix decomposition algorithm is used to obtain the latent patterns and similarities in the data to obtain a factor matrix, the missing values are calculated using the factor matrix, and the preliminary completed data is secondarily completed according to the missing values, and a collaborative filtering algorithm is used to predict the missing numerical values in the secondarily completed data according to the historical data of similar users or items to obtain collaborative filtering completed data;

[0038] According to the obtained collaborative filtering completed data, a deep learning-based generative model is trained, the trained generative model is used to generate the values of the missing attributes, and generative model completed data is obtained, the completed data is represented as a graph structure, a graph neural network is used for modeling and learning, the existence probability of the missing node attributes or edges is inferred through message passing and aggregation operations, and finally the completed graph structure data is obtained, thereby realizing intelligent data repair and completion.

[0039] In an optional embodiment,

[0040] The method further includes:

[0041] A causal discovery algorithm is used to learn the causal relationship between variables from the data, a causal graph model is constructed, the parent nodes and child nodes of the missing value nodes are determined based on the constructed causal graph, and the dependency relationship between the missing values and the causally related variables is established;

[0042] A conditional probability distribution model is used to calculate the conditional probability distribution of the missing value nodes given the values of the parent nodes, and a preliminary repair and completion result of the missing values is generated according to the estimated conditional probability distribution.

[0043] A corresponding simulation scenario is constructed for the preliminary repair and completion result, in the simulation scenario, an adaptive model selection and parameter tuning mechanism is designed, according to data characteristics and task requirements, a causal discovery algorithm and a causal effect estimation method are automatically selected, and a causal graph model and the selected causal effect estimation method are used to estimate the causal effect of the preliminary repair and completion result on a key variable;

[0044] According to the estimation result of the causal effect, the repair and completion scheme is optimized, and the scheme with the optimal causal effect is selected as the final repair and completion result.

[0045] The second aspect of the embodiment of the application,

[0046] A relational database backup and recovery system based on big data is provided, comprising:

[0047] A first unit is configured to acquire full backup data and incremental backup data of a relational database, intelligently classify the full backup data and the incremental backup data by using a machine learning algorithm, construct a data dependency graph based on the classification result, analyze the correlation and integrity constraint between data, analyze the data dependency graph by using a graph neural network, identify changes in the data structure, and automatically generate a data merging strategy according to the identification result, intelligently merge and process the full backup data and the incremental backup data according to the data merging strategy by using a parallel computing framework, and obtain merged backup data.

[0048] A second unit is configured to automatically select a compression algorithm combination by using an adaptive optimization algorithm based on the features and access patterns of the merged backup data, perform hierarchical compression on the merged backup data to obtain compressed backup data, store the compressed backup data in a distributed storage system, optimize data block distribution, predict the life cycle of the compressed backup data stored in the distributed storage system, realize cold and hot layering of data, and integrate the data dependency graph, the classification result and the cold and hot layering information into a multi-dimensional index structure.

[0049] A third unit is configured to quickly locate target backup data by using the multi-dimensional index structure when it is necessary to restore the relational database to a specified time point, decompress the target backup data by using a parallel decompression technology corresponding to the compression algorithm combination to obtain restored backup data, and intelligently check the restored backup data by using data integrity constraint rules; if the intelligent check is passed, the restored backup data is imported into the relational database to complete data restoration at the specified time point; if the intelligent check is not passed, intelligent data repair and completion are performed, and then verification, checking and import are performed again until the intelligent check is passed.

[0050] A third aspect of the embodiments of the present application,

[0051] An electronic device is provided, comprising:

[0052] A processor;

[0053] A memory for storing processor-executable instructions;

[0054] The processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0055] A fourth aspect of the embodiments of the present application,

[0056] A computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0057] In the present embodiment, the backup data is intelligently classified by a machine learning algorithm, and the data dependency relationship is analyzed using a graph neural network to automatically generate an optimal data merging strategy. Combined with a parallel computing framework, efficient data merging and processing are realized, and the backup recovery efficiency is significantly improved. An adaptive optimization algorithm is used to automatically select the optimal compression algorithm combination, and the merged backup data is hierarchically compressed to effectively reduce the data storage space occupation. Combined with the data block optimization distribution of the distributed storage system, the storage cost is further reduced. A multi-dimensional index structure is constructed, and the data dependency graph, classification results and hot and cold layering information are integrated into the index to realize fast positioning and access of backup data. Parallel decompression technology corresponding to the compression algorithm is used to speed up the data recovery process. The recovered backup data is intelligently checked using data integrity constraint rules to ensure the correctness and consistency of the recovered data. For the case where the check fails, intelligent data repair and completion technology is used to automatically repair the data to ensure the integrity of the data. Based on the access mode and characteristics of the data, the life cycle of the backup data is predicted and the hot and cold data is layered to optimize the data storage and access strategy. Support for restoring to a specified time point meets the data recovery needs of different business scenarios. A distributed storage architecture is used to support horizontal expansion and elastic scaling of backup data to adapt to the growing data volume and high concurrency access requirements. Through the parallel computing framework and multi-dimensional index structure, the backup recovery process is highly scalable. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 The flowchart of the relational database backup and recovery method based on big data of the embodiments of the present application is shown;

[0059] Figure 2 The structure diagram of the relational database backup and recovery system based on big data of the embodiments of the present application is shown;

[0060] Figure 3 A schematic diagram of the framework of the present application solution. DETAILED DESCRIPTION

[0061] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0062] The technical solutions of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and some embodiments may not be described again for the same or similar concepts or processes.

[0063] Figure 1 A flowchart of the method for backup and recovery of a relational database based on big data according to an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the method comprises the following steps. Figure 1

[0064] S101. Obtain full backup data and incremental backup data of a relational database, use a machine learning algorithm to intelligently classify the full backup data and the incremental backup data, construct a data dependency graph based on the classification results, analyze the correlation and integrity constraints between the data, use a graph neural network to analyze the data dependency graph, identify changes in the data structure, and automatically generate a data merging strategy according to the identification results, use a parallel computing framework to intelligently merge and process the full backup data and the incremental backup data according to the data merging strategy, and obtain merged backup data.

[0065] In an optional embodiment,

[0066] Based on the classification results, a data dependency graph is constructed, the correlation and integrity constraints between the data are analyzed, a graph neural network is used to analyze the data dependency graph, changes in the data structure are identified, and a data merging strategy is automatically generated according to the identification results, a parallel computing framework is used to intelligently merge and process the full backup data and the incremental backup data according to the data merging strategy, and the merged backup data is obtained, comprising:

[0067] Based on the classification results, the primary key and unique key fields of each table are identified, these fields are compared one by one with all the fields of all other tables in the database, potential foreign key relationships are identified by comparing the similarity of field names, data types, lengths and value ranges, the data overlap and distribution similarity between the identified potential related fields are calculated, and the correlation strength between the fields is determined;

[0068] ​According to the identified foreign key relationship and field correlation strength, a data dependency graph is constructed, wherein nodes represent data tables and edges represent relationships between tables, the constructed data dependency graph is input into a graph neural network model, through multi-layer graph convolution and pooling operations, combined with an attention mechanism, embedding vectors of each data table node are generated;

[0069] The cosine similarity of the embedding vector of the current timestamp node and the embedding vector of the historical timestamp node is calculated, the change of the data table is judged according to a preset similarity threshold, new tables, tables with data growth exceeding a preset threshold and large-scale data update events are identified, and a data structure change recognition result is generated;

[0070] According to the data structure change recognition result, a data merging strategy is automatically generated, based on the data merging strategy, the backup data is divided into independent processing units according to data tables, a directed acyclic graph of the merging task is constructed, a distributed computing framework is used, a fault tolerance and retry mechanism is set, parallel merging operations are performed, and final merged backup data is generated.

[0071] Illustratively, first, the association between tables is identified by analyzing the metadata information of the data tables, and a data dependency graph is constructed. Among them, by comparing the similarity of field name, data type, length and value range, potential foreign key relationships are identified, and the data overlap and distribution similarity between fields are calculated to determine the correlation strength between fields. The nodes of the data dependency graph represent data tables, and the edges represent the relationships between tables, embodying the dependency and association between tables in the database.

[0072] Next, the constructed data dependency graph is input into a graph neural network model for processing. Graph neural network is a deep learning model specially used for processing graph structure data, which can learn the relationship between nodes and the feature representation of nodes by encoding and propagating nodes and edges in the graph. In this method, multi-layer graph convolution and pooling operations are used for feature extraction and dimension reduction of the data dependency graph, the neighborhood information of nodes is aggregated through convolution operation, and feature compression and abstraction are realized through pooling operation. At the same time, the attention mechanism is introduced, the weights of node features are adaptively adjusted according to the correlation and importance between nodes, and the influence of key nodes is highlighted. After multi-layer graph convolution and pooling operations, and the attention mechanism, the embedding vectors of each data table node are finally generated, representing the position and role of the table in the data dependency graph.

[0073] To identify the changes in the data structure, the concept of timestamp is introduced. By comparing the cosine similarity of the embedding vectors of the current timestamp node and the embedding vectors of the historical timestamp nodes, it can be determined whether the data table has undergone significant changes. Cosine similarity is a common vector similarity measurement method, which measures the similarity between two vectors by calculating the cosine of the included angle. If the cosine similarity between the embedding vector of the current timestamp node and the embedding vector of the historical timestamp node is lower than the preset threshold, it is considered that the data table has undergone significant changes. By setting a suitable similarity threshold, the sensitivity of the change detection can be flexibly controlled. Based on the results of the similarity comparison, newly added tables, tables with data growth exceeding the preset threshold, and events with large-scale data updates can be identified, and the data structure change identification results can be generated.

[0074] Based on the data structure change identification results, a data merging strategy is automatically generated. The data merging strategy refers to how to merge the backup data of different timestamps to ensure the integrity and consistency of the merged data. By analyzing the dependency relationships and changes between data tables, the order and manner of the merging operation can be determined. For example, for newly added tables, they need to be directly added to the merged data set; for tables with data growth exceeding the preset threshold, the incremental data needs to be merged with the original data; for tables with large-scale data updates, the new data version needs to replace the original data version.

[0075] When performing the data merging operation, a parallel processing strategy is adopted, which divides the backup data into independent processing units according to the data tables, and constructs a directed acyclic graph (DAG) of the merging tasks. Each data table corresponds to a merging task, and the dependency relationship between tasks is determined according to the association relationship between data tables. By constructing the DAG of the merging tasks, the execution order and dependency relationship of the tasks can be clearly defined, ensuring the correctness and efficiency of the merging operation.

[0076] To accelerate the data merging process, this method uses a distributed computing framework to distribute the merging tasks to multiple computing nodes for parallel execution. Distributed computing frameworks such as Apache Spark and Apache Flink provide convenient data parallel processing and task scheduling functions, which can fully utilize cluster resources and improve the efficiency of the merging operation. At the same time, in order to ensure the reliability of the merging operation, this method sets up a fault tolerance and retry mechanism. When executing the merging tasks, if abnormal situations such as node failure or network interruption occur, the system will automatically reschedule the failed tasks to other available computing nodes for retry until the task is successfully completed. Through the fault tolerance and retry mechanism, the success rate of the merging operation can be maximized, avoiding data loss or inconsistency problems.

[0077] In this embodiment, the association between data tables is automatically identified, a data dependency graph is constructed, and the structural characteristics of the database are accurately described. The data dependency graph is learned using a graph neural network to generate embedding vectors representing the characteristics of each data table, providing support for subsequent change identification and merging strategies. By comparing the table node embedding vectors of different timestamps, the changes in the data structure are identified, including new tables, tables with a significant increase in data volume, and large-scale data update events, providing a basis for developing a merging strategy. The data merging strategy is automatically generated, and the order and method of the merging operation are determined based on the changes in the data tables and the dependency relationships, ensuring the integrity and consistency of the merged data. A parallel processing and distributed computing framework is used to improve the efficiency and scalability of data merging, enabling the processing of large-scale backup datasets. A fault-tolerant and retry mechanism is set up to ensure the reliability of the merging operation and avoid data loss or inconsistency caused by abnormal situations.

[0078] In an alternative embodiment,

[0079] A directed acyclic graph of the merging task is constructed, and a distributed computing framework is used, with a fault-tolerant and retry mechanism set up to perform parallel merging operations and generate the final merged backup data, including:

[0080] In the distributed computing framework, the data merging task is divided into multiple independent functions according to the pre-set serverless computing model, where each function corresponds to a node in the directed acyclic graph. The functions are deployed to a serverless platform, function trigger configurations are generated based on the edge relationships in the directed acyclic graph, and task scheduling plans are generated based on the priority of function execution.

[0081] According to the task scheduling plan and function trigger configuration, the serverless platform and message queue service are integrated to create a data transmission channel. The object storage service of the serverless platform is used to establish a result storage library for storing intermediate results of function execution. The monitoring tools of the serverless platform are used to collect function call times, execution time, memory usage, and error rate to generate performance reports. Based on the performance reports and the directed acyclic graph structure, function optimization configurations including memory allocation and timeout time are created.

[0082] According to the function optimization configuration, the number of retries and error handling rules for function execution are set to generate a fault-tolerant solution. The distributed characteristics of the serverless platform and the parallel execution paths of the directed acyclic graph are used to create a parallel processing framework for simultaneously executing multiple independent functions.

[0083] Based on the parallel processing framework, function optimization configuration and fault tolerance scheme, the functions are executed according to the priority order in the task scheduling plan, after the execution of the functions corresponding to all nodes of the directed acyclic graph is completed, the final merging function is triggered, the final merging function reads all intermediate results, and the final merged backup data is generated after integration processing.

[0084] Illustratively, first, according to the pre-set serverless computing model, the data merging task is divided into several independent functions. Each function corresponds to a node in the directed acyclic graph, representing an atomic operation or processing step in the data merging process. By reasonably dividing the granularity and responsibility of the function, the modularity and decoupling of the task can be realized, and the flexibility and maintainability of the system can be improved.

[0085] Next, the divided functions are deployed on the serverless computing platform. The serverless computing platform is a computing model based on event-driven and function as a service (FaaS), which allows developers to encapsulate application logic as independent functions and automatically manage the execution environment and resource allocation of functions by the platform. By deploying functions to the serverless platform, automatic scaling and on-demand execution of functions can be realized, reducing the operation and maintenance cost and management complexity of the system.

[0086] At the same time of deploying the functions, function trigger configuration is generated according to the edge relationship in the directed acyclic graph. The function trigger configuration defines the calling relationship and trigger condition between functions, ensuring that the functions are executed in the correct order and dependency relationship. By analyzing the topology structure of the directed acyclic graph, the function trigger configuration can be automatically generated, simplifying the workload of configuration management.

[0087] In order to optimize the execution efficiency and resource utilization of the functions, the task scheduling plan is introduced in this method. The task scheduling plan determines the scheduling order and execution time of the functions according to the execution priority and dependency relationship of the functions. By reasonably arranging the task scheduling plan, the concurrency and throughput of the functions can be maximized, and the waiting time and resource waste of the task can be reduced.

[0088] In the actual execution of the data merging task, the message queue service and object storage service provided by the serverless platform are used to build data transmission channels and result repositories. The message queue service is used to transfer data and trigger signals between functions, ensuring the reliable flow of data between functions. The object storage service is used to store the intermediate results generated during the execution of the functions, providing high availability and high durability data storage capabilities. By integrating the message queue and object storage services of the serverless platform, data sharing and collaboration between functions can be realized, improving the efficiency and reliability of data merging.

[0089] To monitor and optimize the execution performance of the functions, the method utilizes the monitoring tools provided by the serverless platform to collect indicators such as the number of function calls, execution time, memory usage, and error rate, and generates a performance report. The performance report reflects the actual execution and resource consumption of the functions, providing a basis for optimizing function configuration. According to the performance report and the structural characteristics of the directed acyclic graph, function optimization configuration containing memory allocation and timeout can be created to dynamically adjust the resource allocation and execution constraints of the functions, improving resource utilization and function execution efficiency.

[0090] Considering the possibility of exceptions and failures during function execution, the method also designs a fault-tolerant scheme. By setting the number of retries and error handling rules for function execution, the fault-tolerance of function execution can be improved, avoiding the failure of the entire task due to individual function exceptions. Meanwhile, by utilizing the distributed characteristics of the serverless platform and the parallel execution paths of the directed acyclic graph, a parallel processing framework is created to allow multiple functions with no dependency to execute simultaneously, fully leveraging the advantages of distributed computing and improving the concurrency and processing speed of data merging.

[0091] Finally, based on the parallel processing framework, function optimization configuration, and fault-tolerant scheme, the functions are executed according to the priority order in the task scheduling plan. After the execution of all functions corresponding to the nodes of the directed acyclic graph is completed, the final merging function is triggered. The final merging function is responsible for reading all intermediate results, integrating and processing them, and generating the final merged backup data. Through the execution of the final merging function, the entire data merging task is completed, ensuring the correctness and integrity of the merging results.

[0092] In this embodiment, by splitting the data merging task into independent functions and utilizing the automatic scaling and on-demand execution characteristics of the serverless computing platform, the computing resources can be dynamically adjusted to adapt to different scales and loads of data merging requirements. By constructing the directed acyclic graph of function execution and the task scheduling plan, automatic generation of function trigger configuration reduces the workload of manual arrangement and scheduling, improving the efficiency and accuracy of task management. By collecting performance indicators of function execution, generating performance reports, and creating function optimization configuration, resource allocation and execution constraints of functions can be dynamically adjusted to improve resource utilization and function execution efficiency. By setting the number of retries and error handling rules for function execution, a fault-tolerant scheme is constructed to improve the stability and reliability of task execution and reduce task failures caused by function exceptions. By creating a parallel processing framework and utilizing the distributed characteristics of the serverless platform and the parallel execution paths of the directed acyclic graph, concurrent execution of functions can be achieved to improve the processing speed and throughput of data merging. Through the execution of the final merging function, all intermediate results are integrated and processed to generate the final merged backup data, ensuring the consistency and availability of the merging results.

[0093] S102. Based on the features and access patterns of the merged backup data, use an adaptive optimization algorithm to automatically select a combination of compression algorithms, perform hierarchical compression on the merged backup data, obtain compressed backup data, store the compressed backup data in a distributed storage system, optimize data block distribution, predict the life cycle of the compressed backup data stored in the distributed storage system, realize cold and hot layering of data, and simultaneously construct a multi-dimensional index structure, integrate data dependency graph, classification results and cold and hot layering information into the multi-dimensional index.

[0094] Among them, by analyzing the merged backup data, the key features of the data are extracted, such as data type, data distribution, data correlation, etc. These features will provide an important basis for the subsequent selection of compression algorithm combination. Data exploration and statistical analysis methods can be used, such as calculating statistical indicators of data (mean, variance, skewness, etc.), drawing data distribution graphs, and analyzing data correlation matrix, etc. Collect and analyze the historical access records of backup data, and mine the patterns and rules of data access, such as access frequency, access time distribution, access data volume, etc. The access pattern analysis results will guide the selection of compression algorithms to optimize data compression and access performance. Data mining and machine learning methods can be used, such as association rule mining, sequential pattern mining, clustering analysis, etc.

[0095] Considering the characteristics and access patterns of backup data, an adaptive optimization algorithm is designed and implemented to automatically select the optimal compression algorithm combination. This algorithm needs to balance compression ratio, compression speed and data access performance. Heuristic optimization, evolutionary computation and other methods can be used, such as genetic algorithm, particle swarm optimization, simulated annealing, etc.

[0096] In the data life cycle, the characteristics and access patterns of backup data may change. To adapt to these changes, a dynamic adjustment mechanism needs to be designed to monitor changes in data characteristics and access patterns in real time, and automatically adjust the compression algorithm combination according to the changes to maintain optimal compression performance. Adaptive control, reinforcement learning and other methods can be used to realize dynamic optimization of compression algorithm combination.

[0097] According to the importance, access frequency and other factors of the data, the merged backup data is divided into multiple levels, such as key data, hot data, ordinary data, etc. Different levels of data use different compression strategies to balance compression ratio and access performance. Data classification and clustering methods can be used, such as K-means clustering, decision tree, support vector machine, etc.

[0098] For different levels of backup data, an adaptive optimization algorithm is used to select the optimal compression algorithm combination to achieve hierarchical compression. Key data uses a high compression ratio and low compression speed algorithm combination, hot data uses a low compression ratio and high compression speed algorithm combination, and ordinary data uses a compromise algorithm combination. Through hierarchical compression, the overall compression ratio is maximized while ensuring the access performance of key data and hot data.

[0099] To ensure the integrity of compressed data, an integrity verification mechanism needs to be introduced during compression, such as calculating the hash value of the data before and after compression, and storing the hash value together with the compressed data. When decompressing the data, the hash value is recalculated and compared with the stored hash value to verify the data integrity. Secure hash algorithms such as SHA-256, MD5, etc. can be used.

[0100] The compressed backup data is stored in a distributed storage system such as Hadoop HDFS, Ceph, etc. Distributed storage systems can provide high reliability, high scalability, and high throughput data storage services. Through data sharding, replica replication, and other mechanisms, the reliability and availability of data are guaranteed.

[0101] According to the data access mode and node performance, the distribution of data blocks on distributed storage nodes is optimized to improve the locality and parallelism of data access. For example, hot data blocks are stored on high-performance nodes, and related data blocks are stored on the same node or adjacent nodes. Data placement algorithms such as greedy algorithm, genetic algorithm, etc. can be used.

[0102] Life cycle prediction is performed on the compressed backup data stored in the distributed storage system to identify cold data and hot data. Cold data is migrated to low-cost, high-capacity storage devices (such as tape libraries), and hot data is retained on high-performance storage devices (such as SSDs) to achieve cold and hot data layering and reduce storage costs. Data life cycle management strategies and data migration techniques can be used. A multi-dimensional index structure is constructed to integrate data dependency graphs, classification results, and cold and hot layering information into the index. Multi-dimensional indexes can quickly locate and retrieve data, supporting complex data queries and analysis operations. Common multi-dimensional index structures include R-tree, KD-tree, octree, etc.

[0103] In an alternative embodiment,

[0104] Life cycle prediction is performed on the compressed backup data stored in the distributed storage system to achieve cold and hot data layering, and a multi-dimensional index structure is constructed, comprising:

[0105] Multi-dimensional life cycle analysis is performed on the compressed backup data to obtain historical access frequency and time distribution information of the data blocks, and a data life cycle prediction model is constructed based on the historical access frequency and time distribution information.

[0106] Using the data lifecycle prediction model, the future access frequency and time distribution of each data block are predicted, and based on the predicted future access frequency and time distribution, multi-level hot and cold data judgment thresholds are set, and based on the multi-level hot and cold data judgment thresholds, each data block is classified into hot data block, warm data block and cold data block;

[0107] A data dependency graph is constructed to record the association relationship between the data blocks, the hot data blocks are allocated to high-performance storage devices, the warm data blocks are allocated to medium-performance storage devices, and the cold data blocks are allocated to low-cost storage devices, and based on the classification results and storage locations of each data block, a hierarchical storage optimization strategy including data prefetching, cache management and data migration is formulated;

[0108] The hierarchical storage optimization strategy is executed, the access frequency, access delay and migration times of each data block are recorded, a performance optimization report is generated, and based on the performance optimization report, the index dimensions and weights of the multi-dimensional index are determined;

[0109] From the compressed backup data, the metadata information is extracted according to the index dimensions and weights, an initial metadata set is generated, the initial metadata set is preprocessed to obtain structured metadata, based on the data distribution characteristics and query patterns of the structured metadata, the performance indicators of different multi-dimensional index structures are calculated, and based on the performance indicators, the optimal multi-dimensional index structure is selected as the final multi-dimensional index structure.

[0110] Illustratively, first, the compressed backup data is subjected to multi-dimensional lifecycle analysis. By collecting and analyzing the historical access frequency and time distribution information of the data blocks, the access patterns and heat variation laws of the data blocks can be understood. Based on these historical information, a data lifecycle prediction model is constructed for predicting the future access frequency and time distribution of the data blocks. The data lifecycle prediction model can use machine learning algorithms such as time series prediction, regression analysis, etc., to learn the laws of data access patterns by training historical data, and to predict future access.

[0111] According to the prediction results of the data lifecycle prediction model, multi-level hot and cold data judgment thresholds are set. The multi-level hot and cold data judgment thresholds are determined according to the predicted access frequency and time distribution of the data blocks, and are used to divide the data blocks into different heat levels. For example, three thresholds can be set, data blocks with predicted access frequency higher than the first threshold are classified as hot data blocks, data blocks with predicted access frequency between the first threshold and the second threshold are classified as warm data blocks, and data blocks with predicted access frequency lower than the second threshold are classified as cold data blocks. By reasonably setting the thresholds, the data can be managed according to the access characteristics and business requirements of the data.

[0112] After determining the hotness level of data blocks, a data dependency graph is constructed to record the association between data blocks. The data dependency graph reflects the logical relationship and access dependency between data blocks, which can help optimize the storage layout and access path of data. By analyzing the data dependency graph, data blocks with high correlation and frequent access can be stored in close proximity, reducing the delay and overhead of data access.

[0113] According to the hotness level and data dependency of data blocks, the data blocks are allocated to different performance storage devices. Hot data blocks need fast response due to high access frequency, so they are allocated to high-performance storage devices such as solid state drives (SSD) or memory. The access frequency of warm data blocks is relatively low, and they can be allocated to medium-performance storage devices such as high-speed hard drives (HDD). The access frequency of cold data blocks is the lowest, and the access speed requirement is not high, so they can be allocated to low-cost storage devices such as tape libraries or object storage. By reasonably allocating storage resources, data access performance can be guaranteed while reducing storage costs.

[0114] In order to further optimize the access efficiency of data, a hierarchical storage optimization strategy is developed, including data prefetching, cache management and data migration. Data prefetching is to load data that may be accessed in advance to higher performance storage layers according to the access mode and association of data, reducing the delay of data access. Cache management is to save frequently accessed data in cache by reasonably setting cache size, cache replacement strategy, etc., to speed up data reading. Data migration is to dynamically adjust the distribution of data between different storage layers according to the access frequency and storage cost of data, and migrate data that is no longer frequently accessed to lower-cost storage layers to save storage space and cost.

[0115] In the process of executing the hierarchical storage optimization strategy, the performance indicators such as access frequency, access delay and migration times of each data block are recorded to generate a performance optimization report. The performance optimization report reflects the effect and improvement space of the hierarchical storage optimization strategy, providing a basis for further optimization. According to the performance optimization report, the index dimensions and weights of multi-dimensional index are determined. Multi-dimensional index is a high-efficiency data retrieval technology that establishes index on multiple attributes or dimensions of data to speed up data query and filtering. The selection of index dimensions and the setting of weights will affect the efficiency and space occupation of index, which needs to be optimized according to the characteristics and query mode of data.

[0116] From the compressed backup data, extract metadata information according to the index dimension and weight, and generate an initial metadata set. Metadata is information that describes the attributes and characteristics of data, such as data identification, timestamp, keyword, etc. By extracting metadata, data blocks that meet the query conditions can be quickly located and filtered, improving the efficiency of data retrieval. Preprocess the initial metadata set, such as cleaning, formatting, standardization, etc., to obtain structured metadata, which facilitates subsequent index construction and query optimization.

[0117] Based on the data distribution characteristics and query patterns of structured metadata, calculate the performance indicators of different multi-dimensional index structures. Multi-dimensional index structures have various types, such as R-tree, K-D tree, octree, etc., which differ in space division, data organization, and query algorithm. By analyzing the data distribution characteristics and query patterns, such as data dimensions, sparsity, query selectivity, etc., the applicability and performance of different index structures can be evaluated. Common performance indicators include index construction time, query response time, space occupation, etc. According to the comparison results of performance indicators, select the optimal multi-dimensional index structure as the final index scheme.

[0118] In an alternative embodiment,

[0119] Based on the data distribution characteristics and query patterns of structured metadata, the formula for calculating the performance indicators of different multi-dimensional index structures is as follows:

[0120] ;

[0121] Where M represents the multi-dimensional index performance indicator, P represents the number of different multi-dimensional indexes, R j represents the query response efficiency of the jth multi-dimensional index, A j represents the storage space occupation of the jth multi-dimensional index, S j represents the dimension number of the jth index, γ represents the weight coefficient, δ represents the attenuation factor, D j represents the update frequency of the jth index, Q represents the number of different queries, T k represents the processing time of the kth query.

[0122] In this embodiment, by cold and hot grading and tiered storage of data, data blocks of different hotness are allocated to suitable storage devices, making full use of storage resources of different performance and cost, and avoiding resource waste. By migrating cold data to low-cost storage devices, the occupation of high-performance storage devices is reduced, and the overall cost of the storage system is significantly reduced. By data prefetching, cache management and other optimization strategies, frequently accessed data blocks are cached in high-performance storage layers, reducing data access delay and improving data read speed. By data lifecycle prediction and dynamic data migration, according to the access mode and business requirements of data, the distribution of data between different storage layers is dynamically adjusted, realizing the adaptability and flexibility of storage management. By constructing multi-dimensional indexes, the data is indexed according to multiple attributes or dimensions, accelerating the query and filtering operations of data, and significantly improving the efficiency of data retrieval. By selecting appropriate multi-dimensional index structures, using the space partitioning and data organization characteristics of the index, complex query requirements such as multiple conditions and range queries are efficiently supported.

[0123] S103. When it is necessary to restore the relational database to a specified time point, the multi-dimensional index structure is used to quickly locate the target backup data, the parallel decompression technology corresponding to the compression algorithm combination is used to decompress the target backup data, the restored backup data is obtained, the data integrity constraint rules are used to intelligently check the restored backup data; if the intelligent check passes, the restored backup data is imported into the relational database, and the data recovery of the specified time point is completed; if the intelligent check does not pass, intelligent data repair and completion are performed, and then verification, checking and import are performed again until the intelligent check passes.

[0124] Firstly, when backing up data, in addition to storing the data itself, time-related metadata information such as backup timestamp, transaction log serial number, etc. needs to be recorded. These metadata information and backup data are used to construct multi-dimensional index structures such as R-tree, KD-tree, etc. The dimensions of the index can include timestamp, database table name, primary key, etc. Through multi-dimensional indexing, the target backup data of the specified time point can be quickly located.

[0125] When data recovery of a specified time point is needed, first, according to the recovery time point, the backup data closest to the time point is retrieved in the multi-dimensional index. Index retrieval can significantly reduce the time overhead of data positioning. After retrieving the target backup data, it needs to be decompressed. Since the backup data is usually compressed by a compression algorithm to save storage space, it needs to be decompressed when restored. Using parallel decompression technology corresponding to the compression algorithm can make full use of computing resources and speed up the decompression process. For example, for block-based compression algorithms such as LZ4, Snappy, etc., the backup data can be divided into multiple independent blocks and decompressed in parallel to improve the throughput of decompression.

[0126] The backup data obtained after decompression needs to be intelligently checked to ensure the integrity and consistency of the data. The recovered backup data is checked using predefined data integrity constraint rules. These constraint rules can include primary key constraints, foreign key constraints, uniqueness constraints, etc. By checking whether the data satisfies these constraint conditions, errors and inconsistencies in the data can be found. The intelligent checking process can be implemented through SQL queries or specialized data verification tools.

[0127] In an optional embodiment,

[0128] If the intelligent check fails, intelligent data repair and completion includes:

[0129] If the intelligent check fails, intelligent data repair is performed, and the integrity constraint rules are used to automatically correct errors in the data to obtain preliminary repaired data. The integrity constraint rules are converted into machine executable repair operations, and a constraint satisfaction problem solving algorithm is used to infer a repair scheme that satisfies the constraint conditions. According to the repair scheme, the search direction of the repair algorithm is guided, and the preliminary repaired data is further optimized to obtain target repaired data.

[0130] The correlation rule mining algorithm is applied to the target repaired data to obtain the dependency relationship between attributes. According to the dependency relationship between attributes and the known attribute values, the missing attribute values are inferred to obtain preliminary completed data.

[0131] The preliminary completed data is represented as a high-dimensional sparse matrix, and a matrix decomposition algorithm is used to obtain the latent patterns and similarities in the data to obtain a factor matrix. The missing values are calculated using the factor matrix, and the preliminary completed data is secondarily completed according to the missing values. Using a collaborative filtering algorithm, the missing numerical values in the secondarily completed data are predicted based on the historical data of similar users or items to obtain collaborative filtering completed data.

[0132] Based on the obtained collaborative filtering completed data, a deep learning-based generative model is trained, and the trained generative model is used to generate missing attribute values to obtain generative model completed data. The completed data is represented as a graph structure, and a graph neural network is used for modeling and learning. Through message passing and aggregation operations, the existence probability of missing node attributes or edges is inferred, and finally the completed graph structure data is obtained, realizing intelligent data repair and completion.

[0133] Exemplarily, first, in the case that the data is not passed by intelligent checking, intelligent data repair is performed. By using pre-defined integrity constraint rules, errors in the data are automatically detected and corrected. The integrity constraint rules refer to conditions or restrictions that the data must satisfy, such as value range, format specification, uniqueness, etc. By converting the integrity constraint rules into machine executable repair operations, using constraint satisfaction problem solving algorithms such as constraint programming, SAT solver, etc., a repair scheme that satisfies the constraint conditions is inferred. The constraint satisfaction problem refers to finding a set of variable values that satisfy all constraints simultaneously under given constraints. According to the inferred repair scheme, the search direction of the repair algorithm is guided, and the preliminary repaired data is optimized to obtain target repair data that meets the integrity constraints.

[0134] Next, the target repair data is applied to the association rule mining algorithm to discover the dependency relationship between the attributes in the data. Association rule mining is a data mining technique used to discover interesting association patterns between data items. By analyzing the dependency relationship between attributes, the value of a missing attribute can be inferred from the known attribute values. For example, if there is an association rule "occupation = teacher -> education = bachelor's degree or above", then for data with occupation as teacher but missing education attribute, the education can be inferred to be bachelor's degree or above according to the rule. Using association rules to infer and fill in the missing attributes, the preliminary completed data is obtained.

[0135] In order to further improve the accuracy of data completion, the preliminary completed data is represented as a high-dimensional sparse matrix. The high-dimensional sparse matrix refers to a matrix with high dimension and most of the elements being zero. Matrix decomposition algorithms such as singular value decomposition (SVD), non-negative matrix factorization (NMF), etc. are used to reduce the dimension and extract features of the high-dimensional sparse matrix, to obtain the latent patterns and similarities in the data, and to obtain the factor matrix. The factor matrix is a low-dimensional dense matrix obtained by matrix decomposition, which captures the main features and structural information of the original matrix. Using the factor matrix, the missing values are calculated, and the preliminary completed data is completed again.

[0136] In order to make full use of the historical data of similar users or items, a collaborative filtering algorithm is used to predict the missing numerical attributes in the twice completed data. Collaborative filtering is a recommendation algorithm based on the similarity of users or items, which predicts the preference or rating of a target user for a certain item by analyzing the similarity between users or items. The users or items in the data are mapped to a low-dimensional space, the similarity between them is calculated, and the missing numerical attributes of the target user or item are predicted according to the historical data of similar users or items, to obtain the collaborative filtering completed data.

[0137] After obtaining the data for collaborative filtering completion, deep learning techniques are further used to generate values ​​for the missing attributes. Deep learning-based generative models, such as generative adversarial networks (GANs) and variational autoencoders (VAEs), are trained to learn the intrinsic distribution and feature representation of the data. Generative models learn the data distribution to generate new data similar to the training data. Using the trained generative model, missing attribute values ​​are generated based on known attributes, resulting in data for the generative model completion.

[0138] Finally, the completed data is represented as a graph structure and modeled using graph neural networks. Graph neural networks are deep learning models suitable for graph-structured data. They learn node feature representations and relationships through message passing and aggregation. Entities in the data are represented as nodes, and relationships between entities as edges, constructing a data graph. Using graph neural networks, the data graph is encoded and reasoned about. Through message passing and aggregation, the probability of missing node attributes or edges is inferred, ultimately yielding the completed graph data.

[0139] In this embodiment, by utilizing integrity constraint rules, errors in the data are automatically detected and corrected, thereby improving the accuracy and consistency of the data. By combining association rule mining, matrix decomposition, collaborative filtering, and deep learning technologies, missing data are intelligently supplemented from multiple dimensions to improve the integrity of the data. Through a progressive repair and completion strategy, the integrity, accuracy, and consistency of the data are gradually improved, and the data quality is comprehensively improved. Through association rule mining and deep learning, hidden patterns, associations, and feature representations in the data are discovered, and the potential value of the data is mined. Graph neural networks are used to model and learn the supplemented data, thereby improving the model's generalization ability for unknown data and enhancing the availability of the data. Through intelligent data repair and completion methods, the workload of manually processing data errors and missing data is reduced, and data processing efficiency is improved.

[0140] In an optional embodiment,

[0141] The method further comprises:

[0142] Use causal discovery algorithms to learn the causal relationship between variables from data, build a causal graph model, determine the parent node and child node of the missing value node based on the constructed causal graph, and establish the dependency relationship between the missing value and the causally related variables;

[0143] Using the conditional probability distribution model, the conditional probability distribution of the missing value node is calculated under the condition of the given parent node value, and the preliminary repair and completion results of the missing value are generated according to the estimated conditional probability distribution;

[0144] corresponding simulation scenarios are constructed for the preliminary repair and completion results, in which an adaptive model selection and parameter tuning mechanism is designed, according to the data characteristics and task requirements, an automatic selection of causal discovery algorithm and causal effect estimation method is performed, and a causal graph model and the selected causal effect estimation method are used to estimate the causal effect of the preliminary repair and completion results on the key variables;

[0145] According to the estimation results of the causal effect, the repair and completion scheme is optimized, and the scheme with the optimal causal effect is selected as the final repair and completion result.

[0146] Exemplarily, first, a causal graph model is constructed by learning the causal relationships between variables from data using a causal discovery algorithm. Causal discovery refers to inferring the causal relationships between variables from observational data, that is, identifying the causal direction and causal strength. Commonly used causal discovery algorithms include PC algorithm based on conditional independence test, GES algorithm based on score search, LiNGAM algorithm based on functional causal model, etc. These algorithms infer the causal direction and causal strength between variables by analyzing the conditional independence, information theory metrics or non-Gaussianity assumptions between variables, and construct a causal graph model. The causal graph model is a directed acyclic graph, in which nodes represent variables, directed edges represent causal relationships, edge directions represent causal directions, and edge weights represent causal strengths.

[0147] Based on the constructed causal graph, the parent nodes and child nodes of the missing value nodes are determined, and the dependency relationship between the missing values and the causal related variables is established. In the causal graph, if a node is the parent node of another node, it means that the former is the direct cause of the latter. If a node is the child node of another node, it means that the former is the direct result of the latter. By analyzing the position and connection relationship of the missing value node in the causal graph, the parent nodes and child nodes of the missing value node can be determined, that is, which variables have direct causal dependency relationship with the missing value node.

[0148] Using a conditional probability distribution model, the conditional probability distribution of the missing value node given the values of the parent nodes is calculated. Conditional probability distribution describes the probability distribution of the target variable given the values of certain variables. For the missing value node, according to the values of its parent nodes, the conditional probability distribution of the missing value node is estimated. Common methods for estimating conditional probability distribution include maximum likelihood estimation, Bayesian estimation, kernel density estimation, etc. According to the estimated conditional probability distribution, the possible values of the missing value node are generated as the preliminary repair and completion results.

[0149] A corresponding simulation scenario is constructed for the preliminary repair and completion results, and an adaptive model selection and parameter tuning mechanism is designed. The simulation scenario refers to constructing a virtual environment similar to the real scenario according to the preliminary repair and completion results, which is used to evaluate the repair and completion effect. The adaptive model selection and parameter tuning mechanism refers to automatically selecting appropriate causal discovery algorithms and causal effect estimation methods according to data characteristics and task requirements, and adjusting the hyperparameters of the algorithms to optimize the model performance. Common model selection methods include cross-validation, information criterion, Bayesian model selection, etc. Parameter tuning methods include grid search, random search, Bayesian optimization, etc.

[0150] In the simulation scenario, the causal graph model and the selected causal effect estimation method are used to estimate the causal effect of the preliminary repair and completion results on the key variables. Causal effect refers to the degree of influence of a variable change on another variable, reflecting the strength of the causal relationship between variables. Common causal effect estimation methods include propensity score matching, causal trees, causal forests, etc. By estimating the causal effect of the preliminary repair and completion results on the key variables, the effectiveness and impact of the repair and completion are evaluated.

[0151] According to the estimation results of the causal effect, the repair and completion scheme is optimized. By comparing the causal effects of different repair and completion schemes on key variables, the scheme with the optimal causal effect is selected as the final repair and completion result. The optimal causal effect means that after repair and completion, the causal effect of the key variable is closest to the real situation, or the impact on the downstream task is most positive. By optimizing the repair and completion scheme, the quality and reliability of data repair and completion are improved.

[0152] In this embodiment, through the causal discovery algorithm, the causal relationship between variables is automatically learned from the data, the causal graph model is constructed, and the internal causal structure and dependency relationship of the data are revealed. Using the conditional probability distribution model, the conditional probability distribution of the missing value node is estimated according to the parent node information of the missing value node, and a reasonable repair and completion result is generated. An adaptive model selection and parameter tuning mechanism is designed to automatically select the optimal causal discovery algorithm and causal effect estimation method according to data characteristics and task requirements, and adjust the hyperparameters to improve the model performance. Using the causal graph model and the causal effect estimation method, the causal effect of the repair and completion result on the key variables is evaluated, and the effectiveness and impact of the repair and completion are quantified. According to the estimation results of the causal effect, the repair and completion scheme is optimized, and the scheme with the optimal causal effect is selected to improve the quality and reliability of data repair and completion. Through causal discovery and causal effect estimation, missing data is automatically repaired and completed, and the integrity, accuracy and consistency of the data are improved, laying a foundation for subsequent data analysis and mining.

[0153] In an optional embodiment, first, according to the data recovery requirements, the scope of data restoration is confirmed, and the data is imported into a temporary table. Then it is judged whether the database table to be restored adopts a database and table splitting strategy. If so, the database and table splitting algorithm rules need to be determined, and the data in the temporary table is cleaned and mapped to the database and table splitting fields of the target table to obtain the temporary table corresponding to the target table and the database and table splitting implementation algorithm. Next, the data recovery task is configured, mainly including the following contents: target data source configuration, target database table information, and table information to be restored. Among them, the target data source configuration includes the host address, port number, username and password of the target data source and other information, which is used to establish a connection with the target database. The target database table information includes the target database name, target data table name and database and table splitting configuration information, which is used to determine the target position and strategy of data recovery. The table information to be restored includes the related information of the data warehouse source table, which is used to identify the data source to be restored. By configuring these information, the mapping relationship between the temporary table and the target table is established, and the data recovery is prepared.

[0154] In actual execution of the data recovery task, the control of data transmission rate also needs to be considered. By judging whether flow limiting is needed, the number of data synchronization threads and the size of data synchronization batch are dynamically adjusted according to the performance of the current target database table, so as to realize the control of data transmission rate. In this way, it can avoid causing too much performance pressure on the target database table, and ensure the stability and reliability of the data recovery process.

[0155] After the data recovery task is started, the system will gradually restore the data in the temporary table to the target database table according to the configured mapping relationship. During the restoration process, the system will record the number of data that has been synchronized in real time, and monitor the state of data synchronization. At the same time, in order to prevent the problem of data inconsistency caused by abnormality in the data recovery process, the system also prepares for data rollback to ensure the integrity and consistency of data.

[0156] In the embodiment, by importing data into a temporary table and cleaning and mapping the data using a sharding algorithm, the data can be efficiently restored from the temporary table to the target library table, improving the efficiency of data recovery. This method can adapt to different sharding strategies, and by configuring the sharding algorithm rules, the data can be restored to the target library table using sharding, improving the flexibility of data recovery. By judging whether flow limiting is needed and dynamically adjusting the number of data synchronization threads and batch size according to the performance of the target library table, the data transmission rate can be effectively controlled, avoiding excessive performance pressure on the target library table, and ensuring the stability and reliability of the data recovery process. During the data restoration process, the number of synchronized data records is recorded in real time and the synchronization state is monitored, and data rollback preparation is done, which can ensure the integrity and consistency of the data restoration process and avoid data inconsistency. By configuring the target data source, target library table information and table information to be restored, the mapping relationship between the temporary table and the target table is established, making the configuration and management of the data recovery task more simple and intuitive, improving the ease of use and maintainability of the system.

[0157] Figure 2 The structure diagram of the relational database backup and recovery system based on big data according to the embodiment of the application is shown in Figure 2 The system comprises:

[0158] A first unit is configured to obtain full backup data and incremental backup data of a relational database, intelligently classify the full backup data and the incremental backup data using a machine learning algorithm, construct a data dependency graph based on the classification results, analyze the correlation and integrity constraints between data, analyze the data dependency graph using a graph neural network, identify changes in the data structure, and automatically generate a data merging strategy according to the identification results, intelligently merge and process the full backup data and the incremental backup data according to the data merging strategy using a parallel computing framework, and obtain merged backup data.

[0159] A second unit is configured to automatically select a compression algorithm combination using an adaptive optimization algorithm based on the features and access patterns of the merged backup data, perform hierarchical compression on the merged backup data to obtain compressed backup data, store the compressed backup data in a distributed storage system, optimize data block distribution, predict the life cycle of the compressed backup data stored in the distributed storage system, realize cold and hot layering of data, and integrate the data dependency graph, the classification results and the cold and hot layering information into a multi-dimensional index structure.

[0160] The third unit is configured to quickly locate target backup data by using a multi-dimensional index structure when it is required to restore the relational database to a specified time point, decompress the target backup data by using a parallel decompression technology corresponding to a compression algorithm combination, obtain restored backup data, and intelligently check the restored backup data by using a data integrity constraint rule; if the intelligent checking is passed, the restored backup data is imported into the relational database to complete data restoration at the specified time point; if the intelligent checking is not passed, intelligent data repair and completion are performed, and then verification, checking and importing are performed again until the intelligent checking is passed.

[0161] Figure 3 For the scheme framework diagram of the present application, as shown in Figure 3 The whole process is divided into three parts: a data backup module, merging and compression, and a big data platform.

[0162] The data backup module includes a full backup module and an incremental backup module, and is responsible for obtaining full data and incremental data of the relational database, respectively.

[0163] Merging and compression: the obtained full backup data and incremental backup data are transmitted to the module for intelligent merging, and then the merged data is compressed by using a hierarchical compression algorithm.

[0164] Big data platform: the compressed backup data is stored in a distributed file system and a multi-dimensional index is constructed. When data restoration is required, the platform quickly locates target data, decompresses the data by using a parallel decompression technology, and performs data checking, repair and import by using a computing engine, and finally completes data restoration at a specified time point.

[0165] The system combines big data technologies such as distributed storage and parallel computing, and can significantly improve the efficiency and reliability of relational database backup and restoration, and realize management of massive data.

[0166] The third aspect of the embodiment of the present application,

[0167] An electronic device is provided, comprising:

[0168] a processor;

[0169] a memory for storing processor-executable instructions;

[0170] The processor is configured to call the instructions stored in the memory to execute the method described above.

[0171] The fourth aspect of the embodiment of the present application,

[0172] A computer readable storage medium is provided, which stores computer program instructions. The computer program instructions are executed by a processor to implement the method.

[0173] The present application can be a method, an apparatus, a system, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for performing various aspects of the present application.

[0174] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A relational database backup and recovery method based on big data, characterized in that: include: Obtain full and incremental backup data of relational databases and intelligently classify them using machine learning algorithms. Based on the classification results, a data dependency graph is constructed, and the correlation and integrity constraints between data are analyzed. The data dependency graph is analyzed using a graph neural network to identify changes in the data structure. A data merging strategy is automatically generated based on the identification results. A parallel computing framework is used to intelligently merge the full backup data and the incremental backup data according to the data merging strategy to obtain the merged backup data, including: based on the classification results, the primary key and unique key fields of each table are identified, and these fields are compared one-to-one with all fields of all other tables in the database. Potential foreign key relationships are identified by comparing the similarity of field names, data types, lengths, and value ranges. The data overlap and distribution similarity between the identified potential related fields are calculated to determine the strength of the correlation between the fields. Based on the identified foreign key relationships and field correlation strengths, a data dependency graph is constructed, where nodes represent data tables and edges represent relationships between tables. This constructed data dependency graph is input into the graph neural network model. Through multi-layer graph convolution and pooling operations combined with an attention mechanism, an embedding vector for each data table node is generated. Calculate the cosine similarity between the embedding vector of the current timestamp node and the embedding vector of the historical timestamp node, judge the changes in the data table based on the preset similarity threshold, identify newly added tables, tables with data volume growth exceeding the preset threshold, and large-scale data update events, and generate data structure change identification results; Based on the results of data structure change identification, a data merging strategy is automatically generated. Based on the data merging strategy, the backup data is divided into independent processing units according to the data table, and a directed acyclic graph of the merging task is constructed. At the same time, a distributed computing framework is used, and fault tolerance and retry mechanisms are set up to perform parallel merging operations to generate the final merged backup data. The merged backup data is hierarchically compressed and stored in a distributed storage system to perform lifecycle prediction, implement hot and cold stratification of data, and simultaneously construct a multidimensional index structure to integrate the data dependency graph, classification results, and hot and cold stratification information into the multidimensional index; When the relational database needs to be restored to a specified time point, the multi-dimensional index structure is used to quickly locate the target backup data, decompress it, and obtain the restored backup data.

2. The method according to claim 1, characterized in that Construct a directed acyclic graph of merging tasks, utilize a distributed computing framework, set up fault tolerance and retry mechanisms, perform parallel merging operations, and generate the final merged backup data, including: In the distributed computing framework, based on the pre-set serverless computing model, the data merging task is split into multiple independent functions. Each function corresponds to a node in a directed acyclic graph. Each function is deployed to the serverless platform, and the function trigger configuration is generated based on the edge relationships in the directed acyclic graph. The task scheduling plan is also generated based on the priority of the function execution. Based on the task scheduling plan and function trigger configuration, the serverless platform is integrated with the message queue service to create a data transmission channel. The serverless platform's object storage service is used to establish a result repository for storing intermediate results of function execution. The serverless platform's monitoring tools are used to collect the number of function calls, execution time, memory usage, and error rate, and generate a performance report. Based on the performance report and the directed acyclic graph structure, a function optimization configuration including memory allocation and timeout is created. According to the function optimization configuration, the number of retries and error handling rules for function execution are set to generate a fault-tolerant solution. By utilizing the distributed characteristics of the serverless platform and the parallel execution path of the directed acyclic graph, a parallel processing framework for simultaneously executing multiple non-dependency functions is created; Based on the parallel processing framework, function optimization configuration and fault-tolerant scheme, functions are executed in the order of priority in the task scheduling plan. After the functions corresponding to all nodes in the directed acyclic graph are executed, the final merge function is triggered. The final merge function reads all intermediate results, integrates them and generates the final merged backup data.

3. The method according to claim 1, characterized in that Perform lifecycle prediction on compressed backup data stored in distributed storage systems, implement hot and cold data stratification, and build a multi-dimensional index structure including: Perform multi-dimensional lifecycle analysis on compressed backup data to obtain historical access frequency and time distribution information of data blocks. Based on this information, a data lifecycle prediction model is constructed. Using the data lifecycle prediction model, predicting the future access frequency and time distribution of each data block, setting multi-level hot and cold data judgment thresholds based on the predicted future access frequency and time distribution, and classifying each data block into a hot data block, a warm data block, and a cold data block based on the multi-level hot and cold data judgment thresholds; Build a data dependency graph to record the relationships between the data blocks, allocate hot data blocks to high-performance storage devices, warm data blocks to medium-performance storage devices, and cold data blocks to low-cost storage devices. Based on the classification results and storage locations of each data block, formulate a tiered storage optimization strategy that includes data prefetching, cache management, and data migration. Execute a tiered storage optimization strategy, record the access frequency, access latency, and migration times of each data block, generate a performance optimization report, and determine the index dimensions and weights of a multidimensional index based on the performance optimization report; Metadata information is extracted from the compressed backup data according to the index dimensions and weights to generate an initial metadata set. The initial metadata set is preprocessed to obtain structured metadata. Based on the data distribution characteristics and query patterns of the structured metadata, performance indicators of different multidimensional index structures are calculated. Based on the performance indicators, the optimal multidimensional index structure is selected as the final multidimensional index structure.

4. The method according to claim 3, characterized in that Based on the data distribution characteristics and query patterns of the structured metadata, the formulas for calculating the performance indicators of different multidimensional index structures are as follows: ； Among them, M represents the multidimensional index performance index, P represents the number of different multidimensional indexes, and R j A represents the query efficiency of the j-th multidimensional index, j Indicates the storage space occupied by the j-th multidimensional index, S j represents the number of dimensions of the j-th index, γ represents the weight coefficient, δ represents the attenuation factor, D j represents the update frequency of the j-th index, Q represents the number of different queries, and T k Indicates the processing time of the k-th query.

5. The method according to claim 1, wherein The merged backup data is hierarchically compressed and stored in a distributed storage system. Lifecycle prediction is performed to achieve hot and cold stratification of data. A multidimensional index structure is constructed to integrate the data dependency graph, classification results, and hot and cold stratification information into the multidimensional index, including: Based on the characteristics and access patterns of the merged backup data, an adaptive optimization algorithm is used to automatically select a compression algorithm combination, and the merged backup data is hierarchically compressed to obtain compressed backup data. The compressed backup data is stored in a distributed storage system, and the data block distribution is optimized. The life cycle of the compressed backup data stored in the distributed storage system is predicted to achieve hot and cold stratification of data. At the same time, a multidimensional index structure is constructed, and the data dependency graph, classification results, and hot and cold stratification information are integrated into the multidimensional index.

6. The method according to claim 1, characterized in that If the intelligent verification fails, intelligent data repair and completion will be performed, including: If the intelligent verification fails, intelligent data repair is performed, and the errors in the data are automatically corrected using the integrity constraint rules to obtain preliminary repaired data. The integrity constraint rules are converted into machine-executable repair operations, and the constraint satisfaction problem solving algorithm is used to infer a repair solution that meets the constraint conditions. Based on the repair solution, the search direction of the repair algorithm is guided, and the preliminary repaired data is optimized to obtain the target repaired data. Apply association rule mining algorithms to the target repair data to obtain the dependency relationships between attributes. Based on the dependency relationships between attributes and the values ​​of known attributes, the values ​​of missing attributes are inferred to obtain preliminary completed data. Representing the initially completed data as a high-dimensional sparse matrix, using a matrix decomposition algorithm to obtain potential patterns and similarities in the data to obtain a factor matrix, using the factor matrix to calculate missing values, performing secondary completion on the initially completed data based on the missing values, and using a collaborative filtering algorithm to predict the missing numerical types in the secondary completed data based on historical data of similar users or projects to obtain collaborative filtering completed data; Based on the collaborative filtering completion data obtained, a deep learning-based generative model is trained, and the trained generative model is used to generate missing attribute values ​​to obtain the generative model completion data. The completed data is represented as a graph structure, and graph neural network modeling and learning are used. Through message passing and aggregation operations, the existence probability of missing node attributes or edges is inferred, and finally the completed graph structure data is obtained to achieve intelligent data repair and completion.

7. The method according to claim 1, characterized in that The method further comprises: Use causal discovery algorithms to learn the causal relationship between variables from data, build a causal graph model, determine the parent node and child node of the missing value node based on the constructed causal graph, and establish the dependency relationship between the missing value and the causally related variables Using the conditional probability distribution model, the conditional probability distribution of missing value nodes is calculated under the given parent node value. The preliminary repair and completion results of missing values ​​are generated based on the estimated conditional probability distribution. Construct corresponding simulation scenarios for the preliminary repair and completion results. In these simulation scenarios, design an adaptive model selection and parameter tuning mechanism to automatically select causal discovery algorithms and causal effect estimation methods based on data characteristics and task requirements. Use the causal graph model and the selected causal effect estimation method to estimate the causal effects of the preliminary repair and completion results on key variables. According to the estimated results of the causal effect, the repair and completion schemes are optimized, and the scheme with the best causal effect is selected as the final repair and completion result.

8. The method according to claim 1, characterized in that The method further comprises: The restored backup data is intelligently verified using data integrity constraint rules; if the intelligent verification passes, the restored backup data is imported into the relational database to complete data recovery at the specified time point; if the intelligent verification fails, intelligent data repair and completion are performed, and then verification, verification and import are performed again until the intelligent verification passes.

9. A big data-based relational database backup and recovery system, used to implement the method according to any one of claims 1 to 8, characterized in that: include: The first unit is used to obtain full backup data and incremental backup data of a relational database, intelligently classify the full backup data and incremental backup data using a machine learning algorithm, construct a data dependency graph based on the classification results, analyze the correlation and integrity constraints between the data, analyze the data dependency graph using a graph neural network, identify changes in the data structure, and automatically generate a data merging strategy based on the identification results. Using a parallel computing framework, the full backup data and incremental backup data are intelligently merged according to the data merging strategy to obtain merged backup data; The second unit is configured to automatically select a compression algorithm combination using an adaptive optimization algorithm based on the characteristics and access patterns of the merged backup data, perform hierarchical compression on the merged backup data to obtain compressed backup data, store the compressed backup data in a distributed storage system, optimize data block distribution, perform lifecycle prediction on the compressed backup data stored in the distributed storage system, implement hot and cold stratification of data, and simultaneously construct a multidimensional index structure to integrate the data dependency graph, classification results, and hot and cold stratification information into the multidimensional index; The third unit is used to quickly locate the target backup data using a multidimensional index structure when it is necessary to restore the relational database to a specified time point, decompress the target backup data using a parallel decompression technology corresponding to the compression algorithm combination, obtain the restored backup data, and perform intelligent verification on the restored backup data using data integrity constraint rules; if the intelligent verification passes, the restored backup data is imported into the relational database to complete the data recovery at the specified time point; if the intelligent verification fails, intelligent data repair and completion are performed, and then verification, verification and import are performed again until the intelligent verification passes.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images