Big Data Disaster Recovery Management System and Big Data Platform Disaster Recovery Methods

Abstract

This application proposes a big data disaster recovery management system and a big data platform disaster recovery method. The system includes: an application-layer disaster recovery service module, a system-layer disaster recovery service module, and a data-layer disaster recovery service module that are interconnected. The application-layer disaster recovery service module is used to perform disaster recovery processing on the application layer of the big data platform; the system-layer disaster recovery service module is used to perform disaster recovery processing on the system layer of the big data platform; and the data-layer disaster recovery service module is used to perform disaster recovery processing on the data layer of the big data platform. This system can achieve complete and orderly data disaster recovery for the big data platform.

Description

Technical Field

[0001] This application relates to the field of big data disaster recovery technology, and in particular to a big data disaster recovery management system and a big data platform disaster recovery method. Background Technology

[0002] With the widespread application of mobile internet, cloud computing, the Internet of Things, and big data technologies, modern society has entered a new era of big data. Mastering big data assets and making intelligent decisions has become crucial for enterprise success. How to manage big data, unlock its value, and ensure it safeguards enterprise development will be a key focus in the future development of information technology. For large industries, data integrity and business continuity are critical indicators of business needs and regulatory requirements; therefore, big data disaster recovery is imperative. Summary of the Invention

[0003] Based on the above requirements, this application proposes a big data disaster recovery management system and a big data platform disaster recovery method, which can realize complete and orderly data disaster recovery for the big data platform.

[0004] A big data disaster recovery management system, the system comprising at least:

[0005] The interconnected application-layer disaster recovery service module, system-layer disaster recovery service module, and data-layer disaster recovery service module;

[0006] The application layer disaster recovery service module is used to perform disaster recovery processing on the application layer of the big data platform.

[0007] The system-level disaster recovery service module is used to perform disaster recovery processing on the system layer of the big data platform.

[0008] The data layer disaster recovery service module is used to perform disaster recovery processing on the data layer of the big data platform.

[0009] Optionally, the system may also include:

[0010] A general disaster recovery service module that is connected to the application layer disaster recovery service module, the system layer disaster recovery service module, and the data layer disaster recovery service module, respectively;

[0011] The general disaster recovery service module manages disaster recovery synchronization tasks and controls disaster recovery functions by calling the application-layer disaster recovery service module, the system-layer disaster recovery service module, and the data-layer disaster recovery service module.

[0012] Optionally, the application-layer disaster recovery service module includes:

[0013] The data development disaster recovery module is used for version synchronization, disaster recovery, and management of application layer code, scripts, and dependency packages;

[0014] The data analysis and visualization module is used for synchronous disaster recovery of application-layer visual report data, as well as synchronous disaster recovery of templates, permissions, and configurations.

[0015] The data service disaster recovery module is used for the synchronous loading of application-layer custom packages and the synchronous disaster recovery of system metadata.

[0016] The Other Application Service Disaster Recovery Module is used to perform disaster recovery for application services in the application layer that are not covered by the above modules.

[0017] Optionally, the system-level disaster recovery service module includes:

[0018] The data management and disaster recovery module is used at least to perform synchronous disaster recovery of metadata of system-level business and technology, as well as data standards and data catalogs;

[0019] The data acquisition and disaster recovery module is used at least to synchronize and recover data acquisition configuration information at the system level.

[0020] The data integration and disaster recovery module is used at least for the synchronization, disaster recovery, and management of data versions at the system level;

[0021] The data computing disaster recovery module is used at least for synchronizing and recovering the computing queues and basic configuration information at the system layer.

[0022] The intelligent scheduling and disaster recovery module is used at least to perform synchronous disaster recovery on the scheduling flow and job configuration at the system level.

[0023] Optionally, the data layer disaster recovery service module includes:

[0024] The data storage layer path / database / table disaster recovery module is used for disaster recovery of data storage paths, databases, and tables in the data layer.

[0025] The path / database / table disaster recovery priority resource management module is used to manage the disaster recovery priority of data storage paths, databases, and tables in the data layer.

[0026] Optionally, when the data storage layer path / database / table disaster recovery module performs disaster recovery on the data layer, it is specifically used for:

[0027] Disaster recovery is performed separately for the raw data, basic data, and analytical data of the data layer.

[0028] Optionally, when the data storage layer path / database / table disaster recovery module performs disaster recovery on the original data of the data layer, it is specifically used for:

[0029] The application layer of the control big data platform writes the raw data of the big data platform's data layer to the backup data center;

[0030] The backup data center is used to store disaster recovery data.

[0031] Optionally, when the data storage layer path / database / table disaster recovery module performs disaster recovery on the basic data and analysis data of the data layer, it is specifically used for:

[0032] A data layer disaster recovery synchronization scheme is adopted to perform disaster recovery on the basic data and analysis data of the data layer.

[0033] Optionally, the disaster recovery general service module includes:

[0034] The disaster recovery console module is used to display the overall management interface for big data disaster recovery, and to design management processes and manage permissions through this interface;

[0035] The disaster recovery task management module is used to manage disaster recovery tasks;

[0036] The disaster recovery anomaly early warning module is used to identify disaster recovery anomalies and issue voice messages when an anomaly is detected.

[0037] The disaster recovery resource management module is used to manage disaster recovery resources.

[0038] The disaster recovery permission management module is used to manage disaster recovery execution permissions;

[0039] The disaster recovery switchover management module is used to manage the switchover between application-layer disaster recovery, system-layer disaster recovery, and data-layer disaster recovery.

[0040] The data import management module is used to manage the import of disaster recovery data into the big data platform;

[0041] The performance indicator analysis module is used to analyze the disaster recovery performance of the big data platform.

[0042] The disaster recovery operation and maintenance management module is used for the operation and maintenance of the big data disaster recovery management system.

[0043] A disaster recovery method for a big data platform, applied to the aforementioned big data disaster recovery management system, includes:

[0044] Receive system backup instructions;

[0045] According to the system backup instructions, the data layer, system layer and application layer of the big data platform are backed up to the backup data center respectively.

[0046] Optionally, the step of backing up the data layer, system layer, and application layer of the big data platform to the backup data center according to the system backup instruction includes:

[0047] The data layer backup service module of the big data disaster recovery management system is invoked to perform disaster recovery processing on the data layer of the big data platform.

[0048] The system-level backup service module of the big data disaster recovery management system is invoked to perform disaster recovery processing on the system layer of the big data platform.

[0049] as well as,

[0050] The application layer backup service module of the big data disaster recovery management system is invoked to perform disaster recovery processing on the application layer of the big data platform.

[0051] The big data disaster recovery management system proposed in this application can perform disaster recovery at the application layer, system layer, and data layer of a big data platform, thereby achieving complete and orderly data disaster recovery for the big data platform. By implementing the above disaster recovery scheme, not only can the integrity of business data of the big data platform be guaranteed through data layer disaster recovery, but the continuity of application and system processes can also be guaranteed through application layer and system layer disaster recovery. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0053] Figure 1 This is a schematic diagram of the structure of a big data disaster recovery management system provided in an embodiment of this application;

[0054] Figure 2 This is a schematic diagram of another big data disaster recovery management system provided in the embodiments of this application;

[0055] Figure 3 This is a detailed structural diagram of the big data disaster recovery management system provided in the embodiments of this application;

[0056] Figure 4 This is a schematic diagram of the disaster recovery process of the application layer disaster recovery service module and the system layer disaster recovery service module provided in the embodiments of this application;

[0057] Figure 5 This is a schematic diagram of the data layer disaster recovery processing of the big data platform provided in the embodiments of this application;

[0058] Figure 6 This is a flowchart illustrating a disaster recovery method for a big data platform provided in an embodiment of this application;

[0059] Figure 7 This is a schematic diagram of disaster recovery for a big data platform provided in an embodiment of this application. Detailed Implementation

[0060] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] This application proposes a big data disaster recovery management system, see [link to relevant documentation]. Figure 1 As shown, the system includes:

[0062] The interconnected application-layer disaster recovery service module 1, system-layer disaster recovery service module 2, and data-layer disaster recovery service module 3;

[0063] The application layer disaster recovery service module 1 is used to perform disaster recovery processing on the application layer of the big data platform.

[0064] The system-level disaster recovery service module 2 is used to perform disaster recovery processing on the system layer of the big data platform.

[0065] The data layer disaster recovery service module 3 is used to perform disaster recovery processing on the data layer of the big data platform.

[0066] Specifically, the big data disaster recovery management system proposed in this application embodiment is equipped with different disaster recovery function modules, which are used to back up each functional level of the big data platform.

[0067] Generally, big data platforms can be broadly divided into application layer, system layer, and data layer based on their different functions.

[0068] The application layer primarily provides storage and computing services such as data services, data analysis, and graph analysis.

[0069] The system layer mainly performs functions such as data acquisition, data integration, data development, analysis, and mining.

[0070] The data layer is mainly used to store various types of data in the big data platform.

[0071] Based on different processing methods, the data types in the data layer can be divided into raw data, basic data, and analytical data. Raw data is the source of data information and can be reprocessed to generate other integrated data and product data, etc. Basic data is obtained from raw data through simple processing and calculation. Analytical data has a longer source chain and needs to be processed and analyzed based on raw data and / or basic data.

[0072] The big data disaster recovery management system proposed in this application embodiment includes an application-layer disaster recovery service module 1 for performing disaster recovery processing on the application layer of the big data platform; a system-layer disaster recovery service module 2 for performing disaster recovery processing on the system layer of the big data platform; and a data-layer disaster recovery service module 3 for performing disaster recovery processing on the data layer of the big data platform.

[0073] Since the data stored in the data layer is mainly used for the operation of the application layer and system layer, the layers of the big data platform are closely connected. To ensure the connectivity between the layers of disaster recovery data, the modules of the big data disaster recovery management system proposed in this application are also interconnected, and the relationships between the modules are the same as the connections between the layers of the big data platform.

[0074] As can be seen from the above description, the big data disaster recovery management system proposed in this application embodiment can perform disaster recovery on the application layer, system layer, and data layer of the big data platform respectively, thereby achieving complete and orderly data disaster recovery for the big data platform. By executing the above disaster recovery scheme, not only can the integrity of the business data of the big data platform be guaranteed through data layer disaster recovery, but the continuity of application and system processes can also be guaranteed through application layer and system layer disaster recovery.

[0075] As a preferred implementation, see [link to relevant documentation]. Figure 2 As shown in the embodiment of this application, the big data disaster recovery management system also includes a disaster recovery general service module 4.

[0076] The general disaster recovery service module 4 manages disaster recovery synchronization tasks and controls disaster recovery functions by calling the application layer disaster recovery service module 1, the system layer disaster recovery service module 2, and the data layer disaster recovery service module 3.

[0077] Specifically, the general disaster recovery service module 4 mainly implements functions such as big data disaster recovery management and scheduling, and disaster recovery drill management. Within this module, a disaster recovery management console is developed and configured to display the overall management interface for big data disaster recovery, allowing for management process design and access control. Secondly, modules for task management, resource management, access control, failover management, and drill management are added to manage disaster recovery synchronization tasks and control disaster recovery functions. Disaster recovery management functions at each level are completed by calling the application-layer disaster recovery service module 1, the system-layer disaster recovery service module 2, and the data-layer disaster recovery service module 3 of the disaster recovery management system.

[0078] The following is combined with Figure 3 As shown, the specific structure and functions of each part of the big data disaster recovery management system proposed in the embodiments of this application are described.

[0079] See Figure 3 As shown, the application layer disaster recovery service module 1 mentioned above includes:

[0080] The data development disaster recovery module 11 is used for version synchronization, disaster recovery, and management of application layer code, scripts, and dependency packages;

[0081] The data analysis and visualization module 12 is used for synchronous disaster recovery of application layer visualization report data, as well as synchronous disaster recovery of templates, permissions, and configurations.

[0082] Data service disaster recovery module 13 is used for synchronous loading of application layer custom packages and synchronous disaster recovery of system metadata.

[0083] Other application service disaster recovery module 14 is used to perform disaster recovery for application services in the application layer that have not been disaster recovered by the above modules.

[0084] Specifically, within the application layer disaster recovery service module 1, it is further subdivided into data development disaster recovery module 11, data analysis and visualization module 12, data service disaster recovery module 13, and other application service disaster recovery modules 14, depending on the disaster recovery object or management object.

[0085] Among them, the data development disaster recovery module 11 is mainly used for version synchronization disaster recovery and management of application layer code, scripts and dependency packages; the data analysis and visualization module 12 is mainly used for the synchronization of application layer visualization report data, as well as the synchronization of application layer templates, permissions and configurations; the data service disaster recovery module 13 is mainly used for the synchronous loading of application layer custom packages and the synchronization of system metadata; the other application service disaster recovery module 14 is responsible for disaster recovery of application services in the application layer that have not been synchronized or disaster recovered by the above modules.

[0086] See Figure 4 As shown, through the synchronization and disaster recovery processes of the various modules in the application layer disaster recovery service module 1, the data of the big data platform application layer data center can be backed up to the backup data center.

[0087] See Figure 3 As shown, the above-mentioned system-level disaster recovery service module 2 includes:

[0088] The data management disaster recovery module 21 is used at least for the synchronous disaster recovery of metadata of system-level business and technology, as well as data standards and data catalogs;

[0089] The data acquisition and disaster recovery module 22 is used at least for synchronizing and recovering the data acquisition configuration information at the system level.

[0090] The data integration and disaster recovery module 23 is used at least for the synchronization, disaster recovery, and management of data versions at the system level;

[0091] The data computing disaster recovery module 24 is used at least for synchronous disaster recovery of the computing queue and basic configuration information at the system layer.

[0092] The intelligent scheduling and disaster recovery module 25 is used at least for synchronous disaster recovery of the scheduling flow and job configuration at the system layer.

[0093] Specifically, the system-level disaster recovery service module 2 is further subdivided into data management disaster recovery module 21, data acquisition disaster recovery module 22, data integration disaster recovery module 23, data computing disaster recovery module 24, and intelligent scheduling disaster recovery module 25.

[0094] Among them, see Figure 4 As shown, the data management disaster recovery service module 21 is mainly used for synchronizing and disaster recovery of system-level business and technical metadata, as well as data standards and data catalogs. It also synchronizes and recovers system-level permissions, maps, data quality, and other information. The data acquisition disaster recovery module 22 is mainly used for synchronizing system-level data acquisition configurations. The data integration disaster recovery module 23 is mainly used for synchronizing system-level ETL code, scripts, and dependency packages. It also synchronizes and manages system-level data versions. The data computing disaster recovery module 24 is mainly used for synchronizing system-level computing queues and basic configuration information. The intelligent scheduling disaster recovery module 25 is mainly used for synchronizing system-level scheduling flows and job configurations, as well as system-level scheduling time and maintenance information.

[0095] After the various modules in the system-level disaster recovery service module 2 perform synchronization, disaster recovery, and other processing on the data or services they are responsible for, the data of the big data platform system-level data center can be backed up to the backup data center.

[0096] In addition, the application layer and system layer flexibly build off-site disaster recovery systems based on the disaster recovery business undertaking objectives and the system's RTO and RPO indicators. For systems with high disaster recovery levels, disaster recovery is deployed in the same proportion. For other important business systems, the disaster recovery data center is configured with resources at a ratio of 80% or 50% to build the system business layer and application layer deployment.

[0097] See Figure 3 As shown, the data layer disaster recovery service module 3 mentioned above includes:

[0098] Data storage layer path / database / table disaster recovery module 31 is used for disaster recovery of data storage paths, databases and data tables in the data layer;

[0099] The path / database / table disaster recovery priority resource management module 32 is used to manage the disaster recovery priority of data storage paths, databases, and data tables in the data layer.

[0100] Among them, the data storage layer path / database / table disaster recovery module 31 is mainly used for disaster recovery of data in the data layer of the big data platform, including disaster recovery of data storage paths, databases, data tables, etc. in the data layer.

[0101] The path / database / table disaster recovery priority resource management module 32 is mainly used to manage the disaster recovery priority of various contents in the data layer.

[0102] The data types in the data layer can be divided into the following three categories based on different processing methods:

[0103] 1. Raw data, mainly including the original full data and incremental data files.

[0104] 2. Basic data: A data table reflecting historical changes in business processes, consisting of raw data that has undergone simple processing.

[0105] 3. Analyze the data. The dimensional data model formed by business modeling can be used to recalculate production based on basic data and raw data.

[0106] Based on the above data type distinctions, when the data storage layer path / database / table disaster recovery module 31 performs disaster recovery on the data layer, it performs disaster recovery on the original data, basic data, and analysis data of the data layer respectively.

[0107] Raw data is the source of data information and can be reprocessed to generate other integrated data and product data, ensuring business continuity in the event of a disaster. To ensure the integrity of raw data and the consistency of data across regions, when the data storage layer path / database / table disaster recovery module 31 performs disaster recovery on the raw data of the data layer, it controls the application layer of the big data platform to directly write the raw data of the big data platform's data layer to the backup data center.

[0108] When the data storage layer path / database / table disaster recovery module 31 performs disaster recovery on the basic data and analysis data of the data layer, the data layer disaster recovery synchronization scheme is adopted to perform disaster recovery on the basic data and analysis data of the data layer.

[0109] Specifically, the basic data is derived from the original data through simple processing and calculation, and a data-layer disaster recovery synchronization scheme is adopted, that is, basic data disaster recovery is performed in sync with the data-layer disaster recovery frequency. When a disaster occurs, new basic data is calculated from the original data, in which the metadata includes the metadata of the corresponding engine, and a system-layer synchronization strategy is adopted to perform near real-time synchronization at the metadata level.

[0110] The data source chain for analysis is relatively long, so the primary disaster recovery strategy is data layer synchronization. For some high-priority data, the synchronization frequency for disaster recovery should be increased, while for data with lower timeliness, the synchronization frequency can be reduced and staggered appropriately. When a disaster occurs, a pre-execution environment is scheduled to start from the original data, run the missing data, and continuously generate basic and analytical data.

[0111] The above data disaster recovery process can be found in [reference needed]. Figure 5 As shown.

[0112] See Figure 3 As shown, the aforementioned disaster recovery general service module 4 includes:

[0113] The disaster recovery console module 41 is used to display the overall management interface for big data disaster recovery, and to design management processes and manage permissions through this interface.

[0114] The disaster recovery task management module 42 is used to manage disaster recovery tasks;

[0115] The disaster recovery anomaly early warning module 43 is used to identify disaster recovery anomalies and issue voice information when an anomaly is detected.

[0116] The disaster recovery resource management module 44 is used to manage disaster recovery resources;

[0117] The disaster recovery permission management module 45 is used to manage disaster recovery execution permissions;

[0118] The disaster recovery switching management module 46 is used to manage the switching between application-layer disaster recovery, system-layer disaster recovery, and data-layer disaster recovery.

[0119] The data import management module 47 is used to manage the import of disaster recovery data into the big data platform;

[0120] Performance indicator analysis module 48 is used to analyze the disaster recovery performance of the big data platform;

[0121] The disaster recovery operation and maintenance management module 49 is used for the operation and maintenance of the big data disaster recovery management system.

[0122] Specifically, in Disaster Recovery General Service Module 4, a disaster recovery management console is developed and set up to display the overall management interface for big data disaster recovery, enabling management process design and access control. Secondly, modules such as task management, resource management, access control, failover management, and drill management are added to manage disaster recovery synchronization tasks and control disaster recovery functions. Disaster recovery management functions at each level are completed by calling the application-layer disaster recovery service module 1, system-layer disaster recovery service module 2, and data-layer disaster recovery service module 3 of the disaster recovery management system.

[0123] Another embodiment of this application also proposes a disaster recovery method for a big data platform, which can be applied to the aforementioned big data platform disaster recovery management system, specifically to the general disaster recovery service module in the management system.

[0124] See Figure 6 As shown, the method includes:

[0125] S101, Receive system backup command.

[0126] The aforementioned system disaster recovery instructions can be instructions sent by the big data platform administrator or management system to the big data platform disaster recovery management system, instructing the big data platform disaster recovery management system to perform disaster recovery on the big data platform.

[0127] S102. According to the system backup instruction, back up the data layer, system layer and application layer of the big data platform to the backup data center respectively.

[0128] Specifically, such as Figure 7 As shown, when performing disaster recovery on the big data platform, the disaster recovery management system backs up the data layer, system layer and application layer of the big data platform respectively, and stores the backup data corresponding to the data layer, system layer and application layer in the backup data center respectively.

[0129] The specific disaster recovery process can be implemented by referring to the functional descriptions of each module of the big data platform disaster recovery management system mentioned above. For example, the backup of the data layer, system layer and application layer of the big data platform can be achieved by calling the various functional modules of the big data platform disaster recovery management system.

[0130] Optionally, when the above method is applied... Figure 3 When the disaster recovery general service module in the big data platform disaster recovery management system is used, according to the system backup instructions, the data layer, system layer, and application layer of the big data platform are backed up to the backup data center, specifically including:

[0131] The data layer backup service module of the big data disaster recovery management system is invoked to perform disaster recovery processing on the data layer of the big data platform.

[0132] The system-level backup service module of the big data disaster recovery management system is invoked to perform disaster recovery processing on the system layer of the big data platform.

[0133] as well as,

[0134] The application layer backup service module of the big data disaster recovery management system is invoked to perform disaster recovery processing on the application layer of the big data platform.

[0135] As can be seen from the above description, the disaster recovery method for big data platforms proposed in this application can perform disaster recovery at the application layer, system layer, and data layer of the big data platform respectively, thereby achieving complete and orderly data disaster recovery for the big data platform. By implementing the above disaster recovery scheme, not only can the integrity of business data of the big data platform be guaranteed through data layer disaster recovery, but the continuity of application and system processes can also be guaranteed through application layer and system layer disaster recovery.

[0136] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0137] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0138] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0139] The modules and sub-modules in the various embodiments of the present application's devices and terminals can be merged, divided, and deleted according to actual needs.

[0140] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0141] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0142] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0143] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0144] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0145] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0146] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A big data disaster recovery management system, characterized in that, The system includes at least: The interconnected application-layer disaster recovery service module, system-layer disaster recovery service module, and data-layer disaster recovery service module; The application layer disaster recovery service module is used to perform disaster recovery processing on the application layer of the big data platform. The system-level disaster recovery service module is used to perform disaster recovery processing on the system layer of the big data platform. The data layer disaster recovery service module is used to perform disaster recovery processing on the data layer of the big data platform; The application-layer disaster recovery service module includes: The data development disaster recovery module is used for version synchronization, disaster recovery, and management of application layer code, scripts, and dependency packages; The data analysis and visualization module is used for synchronous disaster recovery of application-layer visual report data, as well as synchronous disaster recovery of templates, permissions, and configurations. The data service disaster recovery module is used for the synchronous loading of application-layer custom packages and the synchronous disaster recovery of system metadata. Other application service disaster recovery module is used to provide disaster recovery for application services in the application layer that are not covered by the above modules. The system-level disaster recovery service module includes: The data management and disaster recovery module is used at least to perform synchronous disaster recovery of metadata of system-level business and technology, as well as data standards and data catalogs; The data acquisition and disaster recovery module is used at least to synchronize and recover data acquisition configuration information at the system level. The data integration and disaster recovery module is used at least for the synchronization, disaster recovery, and management of data versions at the system level; The data computing disaster recovery module is used at least for synchronizing and recovering the computing queues and basic configuration information at the system layer. The intelligent scheduling and disaster recovery module is used at least to perform synchronous disaster recovery on the system-level scheduling flow and job configuration. The data layer disaster recovery service module includes: The data storage layer path / database / table disaster recovery module is used for disaster recovery of data storage paths, databases, and tables in the data layer. The path / database / table disaster recovery priority resource management module is used to manage the disaster recovery priority of data storage paths, databases, and tables in the data layer.

2. The system according to claim 1, characterized in that, Also includes: A general disaster recovery service module that is connected to the application layer disaster recovery service module, the system layer disaster recovery service module, and the data layer disaster recovery service module, respectively; The general disaster recovery service module manages disaster recovery synchronization tasks and controls disaster recovery functions by calling the application-layer disaster recovery service module, the system-layer disaster recovery service module, and the data-layer disaster recovery service module.

3. The system according to claim 1, characterized in that, When the data storage layer path / database / table disaster recovery module performs disaster recovery on the data layer, it is specifically used for: Disaster recovery is performed separately for the raw data, basic data, and analytical data of the data layer.

4. The system according to claim 3, characterized in that, When the data storage layer path / database / table disaster recovery module performs disaster recovery on the original data of the data layer, it is specifically used for: The application layer of the control big data platform writes the raw data of the big data platform's data layer to the backup data center; The backup data center is used to store disaster recovery data.

5. The system according to claim 3, characterized in that, When the data storage layer path / database / table disaster recovery module performs disaster recovery on the basic data and analysis data of the data layer, it is specifically used for: A data layer disaster recovery synchronization scheme is adopted to perform disaster recovery on the basic data and analysis data of the data layer.

6. The system according to claim 2, characterized in that, The disaster recovery general service module includes: The disaster recovery console module is used to display the overall management interface for big data disaster recovery, and to design management processes and manage permissions through this interface; The disaster recovery task management module is used to manage disaster recovery tasks; The disaster recovery anomaly early warning module is used to identify disaster recovery anomalies and issue voice messages when an anomaly is detected. The disaster recovery resource management module is used to manage disaster recovery resources. The disaster recovery permission management module is used to manage disaster recovery execution permissions; The disaster recovery switchover management module is used to manage the switchover between application-layer disaster recovery, system-layer disaster recovery, and data-layer disaster recovery. The data import management module is used to manage the import of disaster recovery data into the big data platform; The performance indicator analysis module is used to analyze the disaster recovery performance of the big data platform. The disaster recovery operation and maintenance management module is used for the operation and maintenance of the big data disaster recovery management system.

7. A disaster recovery method for a big data platform, characterized in that, The method, applied to the big data disaster recovery management system according to any one of claims 1 to 6, comprises: Receive system backup instructions; According to the system backup instructions, the data layer, system layer and application layer of the big data platform are backed up to the backup data center respectively.

8. The method according to claim 7, characterized in that, The step of backing up the data layer, system layer, and application layer of the big data platform to the backup data center according to the system backup instruction includes: The data layer backup service module of the big data disaster recovery management system is invoked to perform disaster recovery processing on the data layer of the big data platform. The system-level backup service module of the big data disaster recovery management system is invoked to perform disaster recovery processing on the system layer of the big data platform. as well as, The application layer backup service module of the big data disaster recovery management system is invoked to perform disaster recovery processing on the application layer of the big data platform.

Images